System and method for enhancing content using brain-state data

ABSTRACT

A computer system or method may be provided for modulating content based on a person&#39;s brainwave data, including modifying presentation of digital content at at least one computing device. The content may also be modulated based on a set of rules maintained by or accessible to the computer system. The content may also be modulated based on user input, including through receipt of a presentation control command that may be processed by the computer system of the present invention to modify presentation of content. Content may also be shared with associated brain state information.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/516,903, filed Jul. 19, 2019, which is is a continuation of U.S.patent application Ser. No. 15/991,906, filed May 29, 2018, which is acontinuation of U.S. patent application Ser. No. 14/096,822, filed Dec.4, 2013 and claims all benefit, including priority, of U.S. ProvisionalPatent Application Ser. No. 61/733,223, filed Dec. 4, 2012, entitledSYSTEM AND METHOD FOR ENHANCING CONTENT USING BRAIN-STATE DATA, theentire contents of each which is incorporated herein by this reference.

FIELD OF THE INVENTION

The present invention relates to bio-signal collection methods, andsystems that utilize bio-signal data. This invention relates moreparticularly to visualization tools and methods.

BACKGROUND OF THE INVENTION

Bio-signals are signals that are generated by biological beings that canbe measured and monitored. Electroencephalographs, galvanometers, andelectrocardiographs are examples of devices that are used to measure andmonitor bio-signals generated by humans.

A human brain generates bio-signals such as electrical patterns, whichmay be measured or monitored using an electroencephalogram (EEG). Theseelectrical patterns, or brainwaves, are measurable by devices such asand EEG. Typically, an EEG will measure brainwaves in an analog form.Then, these brainwaves may be analyzed either in their original analogform or in a digital form after an analog to digital conversion.

Measuring and analyzing bio-signals such as brainwave patterns can havea variety of practical applications. For example, brain computerinterfaces (BCI) have been developed that allow users to control devicesand computers using brainwave signals.

Various platforms exist for creating or capturing content; andcommunicating this content either to the creator or the creator'saudience.

There is a need to improve the evocative nature or engaging nature ofcommunication platforms and methods.

SUMMARY OF THE INVENTION

A computer system or method may be provided for modulating content basedon a person's brainwave data. The content may also be modulated based ona set of rules maintained by or accessible to the computer system. Thecontent may also be modulated based on user input, including throughreceipt of a presentation control command that may be processed by thecomputer system of the present invention to modify presentation ofcontent.

In accordance with an aspect of the present invention, there is provideda system comprising: at least one computing device; at least onebiological-signal (bio-signal) sensor in communication with the at leastone computing device; at least one user input device in communicationwith the at least one computing device; the at least one computingdevice configured to: present digital content at the at least onecomputing device for presentation to at least one user; receivebio-signal data of the at least one user from the at least onebio-signal sensor, at least one of the at least one bio-signal sensorcomprising a brainwave sensor, and the received bio-signal datacomprising at least brainwave data of the at least one user; and modifypresentation of the digital content at the at least one computing devicebased at least partly on the received bio-signal data, at least onepresentation modification rule associated with the presented digitalcontent, and at least one presentation control command received from theat least one user input device. The presentation modification rule maybe derived from a profile which can exist locally on the at least onecomputing device or on a remote computer server or servers, which mayco-operate to implement a cloud platform. The profile may beuser-specific. The user profile may include historical bio-signal data,analyzed and classified bio-signal data, and user demographicinformation and preferences. Accordingly, the user profile may representor comprise a bio-signal interaction classification profile.

The digital content presentation may be audio, video, or any other typeof physical, tactile, or any other type of presentation. Modification ofthe presentation of the digital content may include changing a sound orappearance of the content, adding other content to the presentation,associating brain state data with the presentation, or sharing thecontent with one or more other users, optionally through a separatecontent sharing service, network, or computer system.

The presentation modifying may include prioritizing, sorting, orfiltering content, or any other types of presentation modification. Thepresentation modifying may be destructive or non-destructive to theoriginal digital content.

The presentation modifying may include controlling operation of at leastone application at the user's local computing device or on anothercomputing device in communication therewith. For example, an applicationthat may be controlled may include a video recorder application to startrecording when a particular brain state or emotion threshold isachieved.

The digital content presented may include live audio or video content.

The received bio-signal data may comprise time-coded bio-signal data,and the digital content presentation may comprise time-codedpresentation data. The at least one computing device may be configuredto synchronize the time-coded bio-signal data to the time-codedpresentation data at least partly by synchronizing time stamps betweenthe time-coded bio-signal data and the time-coded presentation data,wherein the time-coded presentation data may be used by the at least onecomputing device to determine at least one presentation state of thepresented digital content at a respective time stamp.

The at least one presentation control command may indicate a targetbrain state, and the presentation modification may comprise presentingdigital content that is associated with the target brain state at the atleast one computing device. Accordingly, the present invention mayprovide for modulating content to encourage a user to achieve aparticular user-selected target brain state. At least one of thepresentation modification rules may be associated with achieving theparticular target brain state and may also be applied to determine adigital content presentation modification. The presentation modificationmay also include presenting positive feedback to reinforce or encouragea user to achieve or maintain a particular brain state. The presentationmodification may also include presenting negative feedback to discouragea user from remaining in a non-target brain state (e.g. play apunishment sound).

Information gathered from bio-signal sensor data and presentationcontrol command input can be used by the system to update presentationmodification rules based on the content, goals, user profile, andactivity of the user.

In accordance with an aspect of the present invention, there is provideda system comprising: at least one computing device; at least onebio-signal sensor in communication with the at least one computingdevice; at least one user input device in communication with the atleast one computing device; the at least one computing device configuredto: present digital content at the at least one computing device forpresentation to at least one user; receive bio-signal data of the atleast one user from the at least one bio-signal sensor, at least one ofthe at least one bio-signal sensor comprising a brainwave sensor, andthe received bio-signal data comprising at least brainwave data of theat least one user; determine at least one brain state of the user inresponse to the presented digital content, the brain state determiningbased at least partly on received bio-signal data; and associate thedetermined brain state with the presented digital content. Thepresentation modification of the digital may be based at least partly onthe at least one determined brain state and at least one presentationmodification rule associated with the presented digital content.Accordingly, a user's brain state, or determined emotion, may beassociated with the presented digital content. The digital content andassociated brain state data may be shared to a content sharing service,such as Facebook or Youtube. The associated brain state may also bepresented in some form with the digital content.

The system may include a content analyzer component which may determinefeatures of the presented digital content through analysis. A determinedcontent feature may be correlated via timestamp to a determined userbrain state. For example, the content analyzer may determine that thereis an advertisement playing and change the way the content is analyzed.

The system may include a content recommendation engine component whichmay use brain state data and analysis from the content analyzercomponent of content to make better recommendations. For example, thecontent analyzer may determine that the user has strong positivereactions to fight scenes in movies. The content recommendation enginemay extrapolate from this determination to recommend another particularmovie featuring fight scenes to the user.

In accordance with another aspect of the present invention, the at leastone computing device may be configured to receive communication contentduring a communication session established over a communications networkwith at least one remote computing device; wherein the digital contentpresenting comprises presenting the received communication content atthe at least one computing device for presentation to at least one user;and the determined brain state associating comprises associating thedetermined brain state with the received communication content.Accordingly, one or more users' respective brain state(s) may be trackedin a chat or other communication session. The at least one computingdevice may be configured to transmit the brain state associated with thereceived communication content to the at least one remote computingdevice over the communications network. In another aspect of the presentinvention, users of a social media or other communication service may bematched to one another based at least partly on similarities identifiedby the present invention in the respective users' brain states.

In accordance with another aspect of the present invention, the at leastone computing device may be configured to modify presentation of thedigital content at the at least one client computing device based atleast partly on the at least one determined brain state and at least onepresentation modification rule associated with the presented digitalcontent. Accordingly, the present invention may provide for modulatingcontent (either the communication content or other digital content) bythe brain state of the user at the computing device.

In accordance with another aspect of the present invention, the at leastone computing device may be configured to modify presentation of thedigital content at the at least one client computing device based atleast partly on the at least one received brain state and at least onepresentation modification rule associated with the presented digitalcontent. Accordingly, the present invention may provide for modulatingcontent at the computing device based on the brain state of the remoteuser participating in the chat or other communication session.

In accordance with an aspect of the present invention, there is provideda system comprising: at least one computing device; at least onebio-signal sensor in communication with the at least one computingdevice; the at least one computing device configured to: receivebio-signal data of at least one user from the at least one bio-signalsensor, at least one of the at least one bio-signal sensor comprising abrainwave sensor, and the received bio-signal data comprising at leastbrainwave data of the at least one user; determine at least one brainstate of the user, the brain state determining based at least partly onreceived bio-signal data; and select digital content for presentation tothe at least one user based on the determined at least one brain stateof the user; present the selected digital content at the at least onecomputing device. Accordingly, the present invention may provide forpresenting digital content (e.g. advertisement content) to a user whenthe user is in a particular brain state (e.g. happiness).

In accordance with an aspect of the present invention, there is provideda system comprising: at least one computing device; at least onebio-signal sensor in communication with the at least one computingdevice; at least one user input device in communication with the atleast one computing device; the at least one computing device configuredto: receive bio-signal data of at least one user from the at least onebio-signal sensor, at least one of the at least one bio-signal sensorcomprising a brainwave sensor, and the received bio-signal datacomprising at least time-coded brainwave data of the at least one user,determine at least one brain state of the user, the brain statedetermining based at least partly on received bio-signal data, thedetermined at least one brain state comprising time-coded brain statedata; recording time-coded user activity data representing at least oneuser interaction with the at least one computing device; synchronize thetime-coded brain state data to the time-coded user activity data atleast partly by synchronizing time stamps between the time-coded brainstate data and the time-coded user activity data; and determine thebrain state of the at least one user at a respective time stamp of therespective user interaction based at least partly on the time-coded useractivity data. Accordingly, the present invention may provide foranalyzing content creation (including brain state data) to generatecontent creation attributes (including identifying stages of contentcreation by time stamps and determining corresponding brain state(s) atthose time stamps).

In accordance with an aspect of the present invention, the at least oneuser interaction may comprise at least one content creation activity bythe user, and the recorded time-coded user activity data represents thestatus of the at least one content creation activity at respective timestamps.

In accordance with an aspect of the present invention, the at least onecomputing device may be configured to modify presentation of digitalcontent associated with the at least one user interaction based at leastpartly on a correspondence between the time-coded brain state data at atleast one time stamp with the time-coded user activity data at the atleast one time stamp. Accordingly, the present invention may provide formodulating presentation of the user-created content based at leastpartly on the user's brain state when creating the content. Instances ofthe user-created content associated with particular time stamp(s) may bemodulated based on application of presentation modification rules,user-inputted presentation control commands, and based on the user'sbrain state data corresponding to the same time stamp(s).

In accordance with an aspect of the present invention, a target brainstate may be associated at the at least one computing device withencouraging at least one content creation activity, and wherein themodifying presentation of digital content comprises presenting digitalcontent that is associated at the at least one computing device with thetarget brain state. Accordingly, the present invention may providefeedback to the user by way of modifying presentation of the digitalcontent in order to encourage creativity or to encourage the user toachieve a particular brain state. The feedback presented may indicatehow close or how far the user is from achieving the target brain state,and may also include recommended actions the user may perform in orderto try to achieve the target brain state.

In accordance with an aspect of the present invention, the system maycomprise at least one user input device in communication with the atleast one computing device; and the target brain state may be based atleast partly on at least one presentation control command received fromthe at least one user input device.

In accordance with an aspect of the present invention, the system maypresent a stimulus like a ping that is time stamped and analyzed todetermine a corresponding evoked response in the user's brain state. Thecorrelation may be performed up to about 2000 ms after presentation ofthe stimulus. Accordingly, the system may be calibrated to betteranalyze the user's particular brain waves.

In accordance with an aspect of the present invention, the system maysynchronize brain state measurements of the user to the content of asecond user. For example, a second user may also be watching the sametelevision show as the first user and a comparison of brain waves mayshow that both are sharing a similar experience. The presentationmodifying of the present invention may provide real-time presentationsof the brain state data of both users, in a chart, graph, icon, or otherindicator. A stream of the content being consumed by the users may alsobe modulated in some way based on the respective users' brain state dataor based on any determined correspondences between the respective users'brain state data.

In accordance with an aspect of the present invention, a computernetwork implemented system enhancing content created using acommunication utility is provided, the system comprising: a biofeedbackcomputer system linked to a bio-signal processing system for capturingand analyzing bio-signal data so as to extract one or more state of mindfeatures related to at least one individual interacting with thebiofeedback computer system; wherein the biofeedback computer system isconfigured to extract the state of mind features continuously and relatethese to creation of the content; and a content enhancement computersystem which when executed: accesses one or more enhancement parametersfor enhancing the content; processes the state of mind features; appliesone or more applicable display rules; generates and makes availableenhancements to the content based on the state of mind features.

A computer system implemented method is provided comprising:

(a) providing a biofeedback computer system linked to a bio-signalprocessing system for capturing and analyzing bio-signal data so as toextract one or more state of mind features related to at least oneindividual;

(b) linking the biofeedback computer system to a content generator suchthat state of mind features are continuously related to the creation ofthe content;

(c) accessing one or more enhancement parameters for enhancing thecontent; and

(d) processing the state of mind features and applying one or moreapplicable display rules so as to generate and makes availableenhancements to the content based on the state of mind features, using auser interface linked to the content processor.

In this respect, before explaining at least one embodiment of theinvention in detail, it is to be understood that the invention is notlimited in its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The invention is capable of otherembodiments and of being practiced and carried out in various ways.Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described, by way of example only, withreference to the attached figures, wherein:

FIG. 1 illustrates an embodiment of the platform, including a possiblecloud service model;

FIG. 2 illustrates a possible output of the present invention, wherefont of text is varied based on brain state information;

FIG. 3 shows a graph that illustrates variation of brain-stateinformation;

FIG. 4 shows a possible implementation of a mood chart in accordancewith the present invention;

FIG. 5 illustrates a tag cloud modulated by the present invention;

FIG. 6 shows a use case scenario of brain state information;

FIG. 7 illustrates a system diagram view of an embodiment of the presentinvention;

FIG. 8 illustrates a system diagram view of an embodiment of the presentinvention;

FIG. 9 illustrates a system diagram view of an embodiment of the presentinvention;

FIG. 10 illustrates a system diagram view of an embodiment of thepresent invention;

FIG. 11 illustrates a system diagram view of an embodiment of thepresent invention;

FIG. 12 illustrates a system diagram view of an embodiment of thepresent invention;

FIG. 13 illustrates a system diagram view of an embodiment of thepresent invention;

FIG. 14 illustrates a system diagram view of an embodiment of thepresent invention;

FIG. 15 illustrates a system diagram view of an embodiment of thepresent invention;

FIG. 16 illustrates a system diagram view of an embodiment of thepresent invention for real-time bidding for online advertising delivery;

FIG. 17 illustrates a system diagram view of an embodiment of thepresent invention for real-time bidding for online advertising deliveryenabled via a sensor application;

FIG. 18 illustrates a system diagram view of an embodiment of thepresent invention for real-time bidding for online advertising deliveryenabled via instantaneous reading of sensors;

FIG. 19 illustrates a system diagram view of an embodiment of thepresent invention for real-time bidding for online advertising deliveryenabled via streaming sensor data requested by browser JavaScript;

FIG. 20 shows an exemplary post to a social media service sharing brainstate data of a user, in accordance with an aspect of the presentinvention;

FIG. 21 shows an exemplary post to a social media service sharing brainstate data of a user associated with a shared content item, inaccordance with an aspect of the present invention;

FIG. 22 shows an exemplary indication of a user's brain state uponcompletion of presentation of a content item, in accordance with anaspect of the present invention;

FIG. 23 shows an shared video locked for viewing until a viewer achievesa particular brain state, in accordance with an aspect of the presentinvention;

FIG. 24 shows an implementation of an aspect of the present invention,where a shared video is being viewed and the viewer's bio-signal data isbeing tracked and recorded;

FIG. 25 shows an implementation of an aspect of the present invention,where photos or other content are grouped and displayed by brain stateinformation tagged thereto;

FIG. 26 shows an implementation of an aspect of the present invention,where a border of a video window is modulated;

FIG. 27 shows an implementation of an aspect of the present invention,where the user's current brain state may be communicated to the user;

FIG. 28 shows exemplary method steps of an aspect of the presentinvention, to share content;

FIG. 29 shows an example of a shared video in accordance with an aspectof the present invention;

FIG. 30 shows an example of a representation of a user commenting on apost on social media service;

FIG. 31 shows exemplary method steps of an aspect of the presentinvention, to share content;

FIG. 32 shows a brain state indicator that may be presented forrespective content items, in accordance with an aspect of the presentinvention;

FIG. 33 shows an implementation of an aspect of the present invention,where a list of content items is sorted by brain state data;

FIG. 34 shows an example of a video conference where a conferenceparticipant's brain state data is also presented, in accordance with anaspect of the present invention;

FIG. 35 shows an example of a video conference where the received brainstate data of a chat participant may be used to overlay a visualrepresentation of the emotion over the video, in accordance with anaspect of the present invention;

FIG. 36 shows comparing a user's brain state response against therespective brain state responses of one or more other users, inaccordance with an aspect of the present invention;

FIG. 37 shows an example of using brain state data with a wearablecomputing device, in accordance with an aspect of the present invention;

FIG. 38 shows changing an alert notification based on the user'sdetermined brain state, in accordance with an aspect of the presentinvention;

FIG. 39 shows an implementation of an aspect of the present invention,where a user's brain state response may be used to modulate presentationof an advertisement;

FIG. 40 shows creation of a text document with associated recorded brainstate data of the document creator, in accordance with an aspect of thepresent invention;

FIG. 41 shows an implementation of an aspect of the present invention,where brain state feedback may be received during a public performance;

FIG. 42 shows playing back recorded live content together with apresentation of brain state responses of audience members, in accordancewith an aspect of the present invention;

FIG. 43 shows modulating presentation of desktop icons in accordancewith an aspect of the present invention;

FIG. 44 shows a list of callers waiting in a call queue together witheach caller's associated brain state, in accordance with an aspect ofthe present invention;

FIG. 45 illustrates a diagram view of a rules engine or modulationcontroller component in accordance with an aspect of the presentinvention;

FIG. 46 illustrates a generic computer used to implement aspects of thepresent invention; and

FIG. 47 illustrates how real-time advertising bidding may beimplemented.

In the drawings, embodiments of the invention are illustrated by way ofexample. It is to be expressly understood that the description anddrawings are only for the purpose of illustration and as an aid tounderstanding, and are not intended as a definition of the limits of theinvention.

DETAILED DESCRIPTION

The present invention consists of a computer implemented method, and acomputer system that uses brain-state data in order to enhance theexpression conveyed using a communication utility. The communicationutility may consist any manner of utility, program, system, tool, orplatform that enables communication of information in any medium.Communication utility may include for example a word processing utility,note taking utility, journaling application, audio or video player, anemail application, an instant messaging application, note takingapplication, an audio recorder, video recorder, a drawing application, asocial networking platform, and a variety of other applications thatenable one or more users to record their thoughts, ideas or expressions(“expression”). The communication utility may generate or accesscontent. The communication utility may also be referred to as a contentprocessor or a content generator.

Brain-state information may be used in this invention to deriveinformation regarding the emotional state or mood of one or more users.While the present disclosure discusses brain-state information inparticular to derive this information, other data may be used instead ofor in addition to brain-state information, while utilizing the contentenhancement features of the present invention.

The computer implemented method and computer system of the presentinvention enables the capture of brain-state data in conjunction withthe use of the communication utility. In one aspect of the invention,the communication utility is linked to, or incorporates, a brain-statedata capture system for logging brain-state data in a way that may besynchronized with the communication utility such that the one or moreuser's brain-state data is associated with the expression stored usingthe communication utility. In one aspect of the invention, thebrain-state data is captured continuously, and in real time.

In another aspect of the invention, the brain-state data capture systemincludes or embodies a series of rules for converting particularbrain-state data into enhancements of the expressions, also referred toas “embellishment” in the present disclosure. These enhancements serve avariety of purposes, and these may depend on the particular type ofcommunication utility, and also on its user.

For example, the embellishment of expression, using the presentinvention may: (a) enable a user to record their emotional context inconnection with their expression; (b) to help the user, or the user'saudience to comprehend the state of mind that was associated with aparticular communication; (c) create content that is more engaging; (d)create content that holds deeper emotive meaning; and (e) create contentthat provokes recollection of other information associated with thecontent, based on the fact that people are visual and the linking ofvisual enhancements linked to state of mind is likely to encode thecontent in a way that permits the user to access memories associatedwith the content. In other words, the present information may permitsusers to encode content with a broader range of memories, and enablethese memories to be more readily recollected at some point in thefuture.

Other implications of the embellishments of the present invention arediscussed below.

In another aspect of the invention, the computer system of the presentinvention also includes an analyzer that analyzes the expression so asto interpret the expression for enabling the embellishment to be made ina coherent way, and in a way that extends the meaning carried by theimpression. The analyzer permits the generation of embellishments thattruly enhance the content. In one implementation of the analyzer, theanalyzer (a) accesses the brain-state data, (b) analyzes the brain-statedata, (c) maps the brain-state data into one or more of a plurality ofmoods or emotional states.

In one aspect of the invention, the brain-state data is capturedcontinuously and mapped to the creation of the content so as to generateover a period of time one or more moods or emotional states, in that themoods or emotional states may vary over the period of time.

FIG. 4 illustrates a possible set of emotional states to which theanalyzer may map brain-state data for a particular user, this casepositive vs. negative, lethargic vs. high energy. Various other moods oremotional states are possible such as happy, sad or bored.

In another aspect of the invention, a user interface is linked to thecommunication utility, and is configured to allow access to the contentin a form that is embellished using the present invention.

In one implementation, the user interface modifies the content based onthe moods using a set of display rules that are evocative of thedifferent moods or emotional states. The display rules may be generatedby a user, or may be established by the system. The display rules mayconsist of a variety of different rules for encoding state of mindinformation into content.

For example, the display rules when executed may adapt content usingbrain-state data in the following ways: by (1) modifying font size oftext as shown in FIG. 2; (2) modifying the fonts used in text wherecertain fonts are understood to indicate different moods; or (3)modifying line weight. Further details regarding possible display rulesare set out below.

Brain-state information may be most evocative if linked to the time ofcreation of the content, however, there could be a separation betweentime of creation and brain-state information. For example, anapplication may be used to generate content, and then an “enhancementmode” may be triggered where a user may consume the content (read,listen to or view the content for example) and then encode the contentwith brain-state based enhancement based on the present invention. Thisseparation of time of creation and brain-state based encoding may bedesirable where the creation of the content does not occur in real timeor near real time for example because content is created iteratively(for example because the creative process is iterative because of theneed for editing for example) and therefore generation of emotionalcontext using brain-state information may not result in meaningfulemotional context to the content.

The computer implemented method and computer system of the presentinvention enables the capture of brain-state data in conjunction withthe use of the communication utility. In one aspect of the invention,the communication utility is linked to, or incorporates, a brain-statedata capture system for logging brain-state data in a way that may besynchronized with the communication utility such that the one or moreuser's brain-state data is associated with the expression stored usingthe communication utility. In one aspect of the invention, thebrain-state data is captured continuously, and in real time.

In another aspect of the invention, the brain-state data capture systemincludes or embodies a series of rules for converting particularbrain-state data into enhancements of the expressions, or embellishingthe expressions. These enhancements serve a variety of purposes, andthese may depend on the particular type of communication utility, andalso on its user.

As previously stated, the embellishment of expression, using the presentinvention may: (1) enable a user to record their emotional context inconnection with their expression; (2) to help the user, or the user'saudience comprehend the state of mind that was associated with aparticular communication; (3) create content that is more engaging; (4)create content that holds deeper emotive meaning. Other implications ofthe embellishments of the present invention are discussed below.

FIG. 4 illustrates a possible set of emotional states to which theanalyzer may map brain-state data for a particular user. For example,the emotional states may include whether a user is happy, sad, bored.The moods generally may be “positive” or “negative”.

In another aspect of the invention, a user interface is linked to thecommunication utility, and is configured to allow access to the contentin a form that is embellished using the present invention.

In one implementation, the user interface modifies the content based onthe moods using a set of display rules that are evocative of thedifferent moods or emotional states.

The present invention helps people remember things by creating contentthat is more layered and evocative based on encoding of mood or state ofmind information. This helps users remember context either at the timeof creation, or afterwards when the embellished content is retrieved.

Computer System

FIG. 1 illustrates one possible computer system embodiment of thepresent invention. In the particular embodiment shown, the analyzer andrules engine referred to below are implemented on the client side.

In the embodiment shown, a server computer (10) is connected to theInternet. The server computer (10) may be implemented as a server farm.The server computer (10) may be linked to a server application (12) ofthe present invention. The server application (12) may be implemented anapplication repository. The computer system may also be implemented as acloud computing service.

The server computer (10) may be linked to one or more client devices(14). A communication utility (16) may be linked to the client device(14). The communication utility (16) also be may be an Internetimplemented platform accessed using the client device (14). In addition,one or more sensors (18) may be associated with a user for capturebrain-state information, or information relevant to brain-state. Thecommunication utility (16) may include, or be linked to, functionalitythat correlates sensor data to particular content, which may beimportant for example for applications where brain-state information iscaptured continuously in order to permit enhancement of content inconjunction with its creation.

In the implementation shown in FIG. 1, the computer system incorporatesa brain-state profile system (20) implemented to the server application.The brain-state profile system (20) may be implemented similar to theone described in PCT Patent Application No. PCT/CA2013/000785, theentirety of which is incorporated herein by reference. A profile manager(22) may capture over time a profile (24) that includes one or moreattributes for best capturing the mood of a particular user. The profilemay also include one or more preferences of the particular user forembellishing their content. Each person's brainwaves are different,therefore requiring slightly different tunings for each user. Also,people learn to control their brain-state over time, and thereforeparameters relevant for creating brain-state profile may change overtime. New parameters can be calculated based on collected data, and canform part of a user's dynamic profile, by operation of the profilemanager.

The computer system may also include a rules engine (26) that includes aseries of rules (28) to convert brain-state information into contentenhancements. The rules engine (26) may be adapted to apply differentrules: (a) depending on the user, and their profile; (b) thecommunication utility (16); (c) preferred mechanisms to enhance contentassociated with (b), based on (a). Various additional mechanisms may beused to enhance content.

The rules engine (26) may be linked to an analyzer (30), which may beimplemented as an analytics engine. The analyzer (30) may be executableto analyze the content linked to the communication utility (16) asdescribed previously. The analyzer (30) may include or link to variouscontent analysis tools, such as a semantic analyzer or a sentimentanalyzer, so as to interpret the meaning of the content to be enhanced.This can assist in the intuitive contextualization of the content usingthe modulated brain-state information.

For example, PCT Patent Application No. PCT/CA2013/000785 suggests apossible mechanism for modulation of signal information, namely signalmodulation, as well as correction of the modulation through“human-in-the loop” feedback control, which yields an error signal thatmay be used to optimize the bio-state estimation, data collection andthen a application of a parameter adaptation method. For example, amaximum likelihood estimator of parameter values based on error signal.

The communication utility (16) can be linked to a user interface (32)which is adapted to enhance content based on brain-state information, asexplained in this disclosure. The user interface (32) may be implementedin the cloud as part of the computer system shown in FIG. 1, where theuser interface (32) connects for example with a communication utility(16) that may be installed on a client device (14), in order toprovision user interface enhancements to the communication utility (16).The communication utility (16) may also be implemented as an Internetutility, and may link to the server computer (10) so as to accesscontent enhancing functionality of the present invention. Acommunication utility (16) may implement one or more features of thepresent invention. Various computer network architectures may also bepossible for implementing the present invention.

The computer system may be linked to sensor device (18) in a variety ofways. For example, a Multi-modal EEG Data-Collection and Adaptive SignalProcessing System (“MED-CASP System”) for enabling single or multi-usermobile brainwave applications may be provided. The MED-CASP System canbe a cross platform mobile EEG platform specifically for enabling BCIapplications that may be adapted to provide the enhancements describedin this disclosure. The MED-CASP system platform may be implemented as ahardware and software solution that is comprised of an EEG headset, aclient side application and a cloud service component. The client sideapplication may be operating on a mobile or desktop computing device.

A particular MED-CASP system implementation may include a range ofdifferent features and functions, for example an EEG headset may bedesigned to target the meditation (such as health and wellness, orhuman-performance) market segment, and may be designed to be usable withother BCI applications. Non-limiting features of this headset mayinclude: an unobtrusive soft-band headset that can be confidently wornin public; and differentiation from prior art consumer EEG solutionsthrough the use of 3, or 4, or more electrodes (rather than one or two).This advancement may enable: estimation of hemispheric asymmetries andthus facilitate measurements of emotional valence (e.g. positive vs.negative emotions); and better signal-to-noise ratio (SNR) for globalmeasurements and thus improved access to high-beta and gamma bands, fastoscillation EEG signals, which may be particularly important foranalyzing cognitive tasks such as memory, learning, and perception. Ithas also been found that gamma bands are an important neural correlateof mediation expertise.

In the same or another non-limiting exemplary implementation, possibleMED-CASP system features may include: uploading brainwaves andassociated sensor and application state data to the cloud from mobileapplication; downloading brainwave & associated data from the cloud;real-time brain-state classification to enable BCI in games or otherapplications; transmitting real-time brain-state data to other userswhen playing a game to enable multi-user games; sharing brainwave datawith other users to enable asynchronous comparisons of results; sharingbrainwave data to other organizations or third party applications andsystems; and support of cloud based user profiles for storing personalinformation, settings and pipeline parameters that have been tuned tooptimize a specific user's experience. In this way, usage of the systemplatform can be device independent.

Each person's brainwaves are different, therefore requiring slightlydifferent tunings for each user. Each person's brain may also learn overtime, requiring the system platform to change algorithm parameters overtime in order to continue to analyze the person's brainwaves. Newparameters may be calculated based on collected data, and may form partof a user's dynamic profile (which may be called bio-signal interactionprofile). This profile may be stored in the cloud, allowing each user tomaintain a single profile across multiple computing devices. Otherfeatures of the same or another non-limiting exemplary implementationmay include: improving algorithms through machine learning applied tocollected data either on-board the client device or on the server;saving EEG data along with application state to allow a machine learningalgorithm to optimize the methods that transform the user's brainwavesinto usable control signals; sharing brainwave data with otherapplications on mobile device through a cloud services web interface;sharing brainwave data with other applications running on client devicesor other devices in the trusted network to provide for the user'sbrainwave data to control or effect other devices; integration of datafrom other devices and synchronization of events with brainwave data aidin context aware analysis as well as storage and future analysis;performing time locked stimulation and analysis to support stimulusentrainment event-related potential (“ERP”) analysis; and dataprioritization that maximizes the amount of useful informationobtainable from an incomplete data download (i.e. data is transmitted inorder of information salience). The core functionality of the MED-CASPsystem may be wrapped as an externally-usable library and API so thatanother developer may use the platform's features in the developer'sapplication(s). The library may be a static library and API for Unity3D,iOS, Android, OSX, Windows, or any other operating system platform. Thesystem platform may also be configured to use a pre-compiled algorithmsupplied by a third party within the library, including the ability fora third party developer using the library, to use the developer's ownalgorithms with the library. The system platform may also supportheadsets from a variety of vendors; personal data security throughencryption; and sharing of un-curated data (optionally usingtime-limited and fidelity limited access) though the sharing ofencryption keys.

Various implementations of the MED-CASP system platform are possible. Inparticular, various different applications may be developed that utilizethe system platform and enable different types of biofeedback systems.

For example the computer system of the present invention may beimplemented as a hardware and software solution that is comprised of anEEG headset, a client side (mobile or PC based) application and a cloudservice component. Various other implementations are possible.

In one implementation, the computer system of the present invention,which may include the MED-CASP system may when executed: (a) receive andlog brainwave & associated sensor data (“brain-state information”), andapplication state data to the cloud; (b) modulate presentation ofdigital content; (c) optionally, associated profile attributes orpreferences are accessed or retrieved; (d) attributes for enhancingcontent associated with a communication utility are constructed(“enhancement attributes”); and (e) the enhancement attributes areexecuted by a communication utility so as to generate content that isenhanced using brain-state information.

The brain-state data capture system (20), linked to the analyzer, whenexecuted establishes in real time or near real time the mood for aparticular user.

Various different display rules may be used to provide a range ofcontent enhancing features.

FIG. 6 illustrates a non-limiting exemplary use case scenario inaccordance with aspects of the present invention. The computer systemmay determine and record one or more emotions associated with a user'smeasured brain state information. Possible benefits of recording auser's motion may include: obtaining insight into the user's interest inthe content being viewed or created by the user; allow a user viewingcontent to receive information about the state-of-mind of the creator ofthe content; to create or fine-tune content to be more engaging to aviewing user, to be more thought provoking to a viewing user, or to bemore easily remembered by a viewing user.

Various embodiments or implementations of the system of the presentinvention are possible. The present invention may be applied tosituations where a user is creating content, or where a user is viewingcontent. In either case, the content may be digital content that can becreated, stored, and viewed on a computing device, such as a desktopcomputer, mobile phone, tablet computer, wearable computer, servercomputer, or across a network of computing devices. The presentinvention may be adapted to any type of computing device, and the use ofthe term “computing device” in the claims is not intended to be limitedto any particular type of computing device. As shown in FIG. 7, a user'sbiological signal information may be acquired and processed by abiological signal processing pipeline of the computer system of thepresent invention. The biological signal processing pipeline may bedefined as in PCT Patent Application No. PCT/CA2013/000785 and may applypreprocessing, feature extraction, and classification to the biologicalsignal information.

The biological signal information, or bio-signal data, may be obtainedfrom the user through the use of at least one bio-signal sensor incommunication with at least one computing device (shown as biologicalsignal acquisition in FIG. 7). The digital content may be provided froma content source, as shown. The digital content may include externalcontent, and may include any one or more of audio, video, text, books,etc. In addition the content source could be biological signal data aswell. The digital content could also be a separate application, such asAdobe Photoshop, or Microsoft Word, which may or may not be executingfrom the computing device at the user. Optionally, the application(s) orother digital content may reside on a separate computer or server, orcloud storage, accessible for viewing, interacting, or otherwiseconsuming or presenting to the user's computing device over acommunications network.

The computing device which the user is using to view, create, orotherwise interact or interface with digital content may or may notnecessarily be the same computing device that processes the bio-signaldata. For example, the user's bio-signal data may be received by a localcomputer, then transmitted over a communications network, such as theInternet, other wide area network, or a local area network, to anothercomputing device for processing. The at least one bio-signal sensor mayinclude at least one brainwave sensor, such as an EEG sensor. Such abrainwave sensor may include at least a pair of electrodes positionableon the user to monitor the user's brainwave activity. The brainwavesensors may be included in a wearable headband as described in thispatent application or in PCT Patent Application No. PCT/CA2013/000785.The computer system may process the bio-signal data using the biologicalsignal processing pipeline shown in FIG. 7, in order to determine atleast one brain state of the user. The brain state data may bedetermined to have occurred in response to user being presented withparticular digital content. The computer system may therefore associatethe user's determined brain state with the presented digital content.

Instead of, or in addition to, performing this association, the computersystem may optionally modify presentation of the digital content at theat least one computing device based at least partly on the receivedbio-signal data and at least one presentation modification ruleassociated with the presented digital content. Application of any suchrules may be applied as shown in FIG. 7 for processing by a modulatorcomponent interfacing with a content source, as shown.

Optionally, the computer system may be further configured to modifypresentation of the digital content based at least partly on at leastone presentation control command received from at least one user inputdevice in communication with the at least one computing device. This maybe represented by the “manual override” shown in FIGS. 8-12 which mayfeed into a modulation controller that controls operation of themodulator. The input device may be a keyboard, touchpad, touchscreen,speech input, mouse, or any other user input device. The user inputdevice may communicate with the at least one computing device throughwired or wireless communications, such as Bluetooth. All or some of thebio-signal sensor(s), user input device(s) and computing device(s) maybe integrated into one wearable computing device or other computingdevice. Presenting digital content for presentation to the user, ormodifying presentation of the digital content may include visualpresentation (such as display on a display device connected to thecomputer system), audio presentation (such as playing sound or music onan audio device connected to the computer system, such as speakers orheadphones), or any other type of presentation (e.g. through any type ofpresentation device or component, such as a tactile feedback device,such as a vibrating game controller or mobile phone). Modulating, ormodifying presentation of the digital content, may include changing howthe digital content appears or sounds, displaying or playing additionaldigital content. Changing appearance of the content may include scalingor moving elements of the digital content on a display, or modifying theopacity, contrast, or colour of particular elements of the digitalcontent. The types of digital content to be displayed or modified is notlimited to any particular content or modifications.

The modulation controller may include or be associated with a set ofrules that are applied to a specific goal established by the user. Themodulation controller may receive features or predictions from thesignal processing pipeline and apply rules that modulate the digitalcontent. In addition, optionally, only parts of the content may bemodulated such as only the audio portion of video. Each time the contentis modulated and presented, the user's response may be determined, andthe content may be modified further in accordance with the rules. Themodulation of the content therefore may therefore be iterative,including based on input from the user. The modification of presentationof the content may include various classes of modification, including:proportional, where there is an amplifier that multiplies the signal bysome constant to increase its amplitude or frequency or some otherfeature; derivative, where the rate of change or acceleration of afeature is used to control the modulator (e.g. if heart rate increasedby 20% then send calming audio to the user); and integral, whereaccumulation or summing up of features used to control the output (e.g.if a user is in a particular brain state for 30 seconds then apply areward tone for neurofeedback).

The modulator may apply rules to the signal as input by the modulationcontroller to control media experienced by the user. For example, in thecase of audio content, volume or equalizer settings, or other propertiesof the audio may be changed depending on the mood of the user. In thecase of neurofeedback, the modulated content could be the speed of acharacter in a game.

In a non-limiting example of the embodiment of the present inventionshown in FIG. 8, the user may be listening to music such that thecontent source is music playing over a sound system that is beingmodulated by the modulator. The user's intention is to have backgroundmusic that has a calming effect allowing the user to focus on a mentaltask and not have the music be a distraction. The music is beingmodulated by analyzing the user's brainwaves and re-modulating the musicif the user's brainwaves indicate agitation or distraction. The user'sbrainwaves are analyzed to determine the user's level of calmness andfocus through acquiring the user's EEG signals using “Biological SignalAcquisition”. The EEG signals are then sent to the Biological SignalProcessing Pipeline which may apply preprocessing, feature extraction,and classification. Preprocessing, feature extraction, andclassification are described in more detail in PCT Patent ApplicationNo. PCT/CA2013/000785. The Modulation Controller may apply a set ofrules to the goal of calm and focus. In this case the ModulationController takes into account: the brain state of the user as determinedby the classifier, their mood goal, and activity. The ModulationController can take into account many other variables including: time ofday, what music came before, what activity is to follow, what activitycalm before, other biological measures etc. In addition, the ModulationController also takes into account user preferences as directlycontrolled by the user by entering commands in the manual override. Theuser may choose to increase volume of the music, skip a track change theequalizer settings for the music. Acoustic effects could be added basedon brain state/emotion. The music may be modulated to adapt to thelistener. The musician may passively express himself or herself byadding characteristics of the musician's brain state to the music. Themusic may become faster/slower, louder/softer, or accompaniments may beadded in accordance with a particular musical composition generatingalgorithm. The music may therefore better express the performer's orcomposer's brain state.

Music may also be recommended based on brain state. For example, musicmay be recommended based on how a user's brain signature responds tocertain kinds of music, and then match/correlate that to how the userwould respond to music that has similar characteristics. A user may alsotag music with a brain state or emotion experienced while creating orplaying the music. The computer system may also be configured tointentionally modulate music or other audio content to assist theaudience in paying attention to the audio content.

The preprocessing described above may include the computer systemprocessing the bio-sensor data. This can involve, as non-limitingexamples, interpolating gaps in signals, noise identification, noisefiltering, spatial filtering, band-pass filtering. Other solutions maybe employed for handling issues related to signal collection andtransmission. Signal processing is a broad area that refers tomathematical methods of extracting or transforming signals. The goal isto pre-process the signal to remove unwanted noise and artifacts or tofocus on a narrow frequency bands of interest. Signal processing mayinclude: filters (frequency response characteristics) (may include: lowpass; high pass; and bandpass); order of filters; artifact detectionmethod (common spatial patterns, Threshold (e.g. log bandpower, signalpower), high frequency EMG with spatial patterns, off); artifact removalmethod (ICA, repair bursts, ignore sections with artifacts); resamplerate; EEG channels to use; electrode referencing scheme (averagereference, specific electrode, linked mastoids); data cleaning (removeflatlines, remove broken channels, remove spikes or bursts); baselineremoval (off or on); and window definition (size and increment).

The feature extraction described above, may include using thepreprocessed data to extract features from the signal. For example, thismay be done through linear projection (such as Short-time Fouriertransform, “STFT”) of the EEG signal or non-linear functions (such asfractal dimension) of the EEG data. The resulting feature space may havegreater or lesser dimensions than the original data. This projection mayaid in the classification of the collected data, for example, byincreasing the separability of classes. Examples feature methods used bythe system (operating on original signal or features of the signal) mayinclude: short time Fourier transform; wavelet; AR model coefficients(auto regressive model of the signal); Non-linear complexity featuressuch as fractal dimensions (Hilbert transform); ensemble empirical modedecomposition; signal derivatives; and regularized covariance matrixestimation (such as ledoit wolfe estimator). Feature extraction is alsoa form of signal processing, however the goal may be to extract featuresthat are useful for machine learning to build prediction models.Frequency domain signal processing may include: fast Fourier transform(“FFT”), wavelet or Hilbert; kernel definition (Hamming window, wavelettype); power spectrum; power per EEG band; power variance per EEG band;peak frequency within a particular band; and phase differences acrosselectrodes within EEG frequency bands. Time domain signal processing mayinclude: window sub-segments; voltage or signal power thresholds; andquantization or discretization thresholds to encode continuous featurevalue to discrete value. Transforming data to extract features may beaccomplished at least by using principal components analysis, lineardiscriminant analysis, eigendecomposition, and/or whiteningtransformations.

In a non-limiting implementation of the present invention, the computersystem may determine at least one brain state of the user, the brainstate being determined based at least partly on received bio-signal dataand select digital content for presentation to the at least one userbased on the determined at least one brain state of the user. In thiscase, the computer system is not necessarily reliant upon digitalcontent having already been displayed to the user. Rather, the computersystem attempts to determine whether the user is in a particular brainstate. If so, the computer system selects digital content forpresentation to the user that is somehow associated with the determinedbrain state, and presents the selected digital content at the at leastone computing device. For example, when a user is in a “happy” brainstate, the computer system may present a particular advertisement to theuser, as showing an advertisement to a happy user may be more effectivethan showing the same advertisement to an angry or distracted user. Whena user is in a “distracted” brain state, the computer system may presenta soothing presentation of content, either by modulating sound to becalming, or by reducing the number of items displayed to the user, or byany other content modulation or modifying method that is configured intothe computer system.

In a non-limiting implementation of the present invention, the receivedbio-signal data may include at least time-coded brainwave data of the atleast one user, and the determined at least one brain state may comprisetime-coded brain state data. The computer system may be configured torecord time-coded user activity data representing at least one userinteraction with the at least one computing device, synchronize thetime-coded brain state data to the time-coded user activity data atleast partly by synchronizing time stamps between the time-coded brainstate data and the time-coded user activity data, and determine thebrain state of the at least one user at a respective time stamp of therespective user interaction based at least partly on the time-coded useractivity data. In this way, the computer system of the present inventionmay determine and record what brain state(s) was exhibited by the userduring particular interactions with the computer system. This may beapplied to situations where the user is playing a video game, chattingwith another user, creating content (e.g. writing text, drawing art,composing music, etc.), or consuming content. The computer system maythen know what the user was feeling at any given time stamp, or moment,of the interaction. The recorded brain state information may be used tomodulate or modify the display of digital content to the user or toother users of the same content. The user's brain state information mayalso be forwarded or displayed in some format to other users viewing orinteracting with the same or similar content.

The computer system of the present invention may present at least asubset of the received bio-signal data to the user. For example, thecomputer system may modify presentation of the digital content by alsopresenting some of the bio-signal data or information derived therefrom.The computer system may determine what of the bio-signal data to displayby analyzing the bio-signal data received. For example, the computersystem may present a visual indication of a user's determined brainstate on a display visible to the user or on a display visible to otherusers. The presentation modifying may include presenting at least onerepresentation of the user's brain state based at least partly on thereceived bio-signal data and the at least one presentation modificationrule. For example, a presentation modification rule may direct thecomputer system to enlarge the size of particular text elementsdisplayed to the user upon determining that the user's brain state isindicative of a lack of focus. The presentation modifying may be furthermodified by at least one presentation control command being receivedfrom a user through an input device connected to or otherwise incommunicating with the computer system or computing device. For example,the presentation control command may include a direction to the computersystem to further increase the size of the text elements, or thepresentation control command may include a direction to the computersystem to revert the size of the text elements to the originallydisplayed size, and to cease to increase size of the text elements inthe future.

In order to correlate received bio-signal data with the presenteddigital content, the computer system may be configured to associate timestamps to the presentation of the digital content at particularintervals, and also to the received bio-signal data. Accordingly, thereceived bio-signal data may include time-coded bio-signal data, and thedigital content presentation may include time-coded presentation data.The computer system may synchronize the time-coded bio-signal data tothe time-coded presentation data at least partly by synchronizing timestamps between the time-coded bio-signal data and the time-codedpresentation data. In this way, the time-coded presentation data may beused to determine at least one presentation state of the presenteddigital content at a respective time stamp. The computer system maymodify presentation of the digital content based on a correspondencebetween the time-coded bio-signal data at at least one time stamp withthe time-coded presentation data at the at least one time stamp.

The time stamps may therefore be used to more precisely determine auser's bio-signal response to the presented digital content. Forexample, a particular instance in time in presentation of anadvertisement to the user that caused the user to be happy or angry,could be determined. The portion of the advertisement causing theparticular bio-signal response, could then be modified to eitheraccentuate or remove the portion of the advertisement.

The computing device may store a bio-signal interaction classificationprofile (as shown in FIG. 9), or the computing device may communicatewith one or more other computing devices in order to access informationstored in such a profile. The profile may interface with the biologicalsignal acquisition, the biological signal processing pipeline, themodulation controller, and the modulator. The bio-signal interactionclassification profile may be user-specific or more generallyapplicable. The profile may include portions that are user-specific andportions that are not user-specific. Each profile may be associated witha particular user. The profile may include information on how todetermine a particular brain state or emotion or the user experience thedigital content presentation. The profile may include directions to thecomputer system on how to modify presentation of the digital contentbased on the received bio-signal data or the determined brain state oremotion of the user. The computer system may also include a data miningimprover, which may interface with the profile, and may learn a user'spreferences and change the rules in the Rules Engine (described ingreater detail later) for achieving the user's goal.

For example, music may be modulated based on prior knowledge of whatworks for the user (e.g. what modulation produces particular brain stateresponses in the user) and is encoded as a set of rules. The rules canbe learned from the user directly training the rules engine, or fromother users as informed through the cloud. In this example, the user hasselected effective working stimulation as their goal. The modulatorcontrols the amount of spatial content that the audio has in orderaffect its pull on the user's awareness. Audio with higher spatialcontent may result in greater engagement in the user. Boundaries are seton the levels of calculated engagement by analyzing brainwaves that areoptimal for the user. When the user crosses a boundary associated withtoo little engagement with the music (or too much engagement with theexternal environment), the spatial content is increased to increaseengagement with the music. If the user becomes too engaged with themusic and thereby working less effectively because the user isdistracted by music, spatial content may be decreased. If the useradjusts the volume, the rules engine may interpret this as a possibilitythat the user requires a different level of engagement. For instance,two rules added to the Rules Engine may include: “Higher volume (thanstandard setting)=greater engagement with music”; and “Lowervolume=lesser engagement of music”. This is a simple example ofmodulation correction. This information can be used to update theboundaries in the rules engine. Different logics are possible for thisupdate.

When the computer system receives a presentation control command inputfrom the user, the computer system may override application of anypresentation modification, and the computer system may further updatethe bio-signal interaction classification profile based at least partlyon the presentation control command. For example, the user may instructthe computer system to perform or not perform a particular presentationmodification, and the user may also instruct the computer system tomaintain this instruction. The instruction may therefore be stored inthe bio-signal interaction classification profile, which may beassociated with the particular user that provided the presentationcontrol command. The presentation control command input may be appliedprior to any other presentation modifying (as shown in FIGS. 9 and 10),or the presentation control command input may be applied subsequent toother presentation modifying (as shown in FIG. 11).

The at least one computing device may be configured to update thebio-signal interaction classification profile based at least partly onbio-signal interaction classification data received from at least onecomputer server over a communications network. The at least onecomputing device may be configured to update the at least onepresentation modification rule based at least partly on presentationmodification rule data received from at least one computer server over acommunications network. Accordingly, various aspects of the computersystem may be stored remotely from the computing device of the user. Inparticular the profile and modulation controller may be stored on acomputer server or servers, at a cloud platform, a shown in FIG. 12. Thedata mining improver may also be implemented in the cloud, as shown, andmay interface with the biological signal processing engine andmodulation controller.

The data mining improver may take advantage of looking across aplurality of user profiles to establish modulation controller rules,algorithms, etc. for demographics that are similar to the present user.Such established rules may then be applied to the present user andstored in the user's profile either in the cloud or on the localcomputing device.

The profile may have a data structure and function as described in PCTPatent Application No. PCT/CA2013/000785, and may store annotated rawbiological data, algorithm settings, preferences and demographics of theuser.

The at least one presentation control command may indicate a targetbrain state for the user to achieve. The presentation modification mayinclude presenting digital content that is associated with the targetbrain state at the at least one computing device. Accordingly, if theuser instructs the computer system that the user wishes to be in arelaxed brain state, or in a happy brain state, the computer system maymodify presentation of the digital content in such a way that isassociated with that targeted brain state, optionally to allow the userto achieve or maintain the targeted brain state.

The presentation modification may include updating an activity feedassociated with the at least one user. For example, where the user isregistered with Facebook, Twitter, or other social media service, thecomputer system may update the user's status or timeline, or make aposting on the service based at least partly on the received bio-signaldata. The computer system may be provided with login access to theuser's respective social media accounts. If the user is determined to beexperiencing a particular brain state or emotion, the computer systemmay incorporate that information into a posting or other type ofsubmission to a social media service. The submission may includeinformation about the activity being performed by the user at the timethe respective bio-signal data was received. Accordingly, the activityfeed updating may include the at least one computing device transmittingat least one activity feed update command to at least one computerserver to update an activity feed associated with the at least one userat the computer server.

Rules Engine (Modulation Controller)

The rules engine, or modulation controller, of the present invention,implemented by the computer system of the present invention maygenerate, store, update, or modify one or more presentation modificationrules that are used by the computer system of the present invention todetermine how to modify presentation of digital content to the user atat least one computing device of the system.

The rules engine, an example architecture of which is shown in FIG. 45,may include a content analyzer component, which may be implemented inhardware or software. The content analyzer may determine events andcharacteristics of the content. The content analyzer may add labelledtimestamps of those events that are synchronized with biological data.The content analyzer may be important as there are many existingtechnologies that can be applied, that are highly optimized and areavailable as software as a service or embeddable library (e.g. voicerecognition). The content analyzer may be implemented using availablecontent analysis techniques, such as more recent auto-encoder and deeplearning methods, as well as commonly used feature and classifiermethods.

Input to the content analyzer may include modulated content, and thecontent analyzer may also receive the original un-modulated content, orinformation from the modulator to improve the performance in situationswhere the modulation may introduce too much data corruption (e.g.captcha). For video salience, the content analyzer may receiveinformation about how other users have reacted to the video, to assistin the estimation of what features the current user may react to. Forapplication state information, the content analyzer may determine whatto analyze based on the state of the application. For example, for aYoutube advertisement, different feature extraction may need to be usedfor the advertisement compared to the content the user is awaiting toview.

Input may also include information about the other user's state in thecase of a synchronous multi user application (e.g. for online chat,where the emotional content is analyzed by a word analyzer that alsoconsiders the other user's available data streams, like level ofdistraction). The user's brain state may be determined to be distracted,and the user may inadvertently use strong language as a result of theuser's attention being divided. By considering other availableinformation, a better estimate of true message content may be produced.

Output from the content analyzer for text content, may include:language/semantic analysis for emotional content; document structuralanalysis; and user state estimation using offline and real-time keyboardentry dynamics.

Output from the content analyzer for video content, may include: scenechange detection in video content (e.g. may use frame differencingmethods, motion tracking estimates; may involve interest pointdetection/description including harris corners, canny edges, or moreadvanced method such as sift features); face detection, and emotionestimation based on facial expression; and interest point detectionbased on other user data (e.g. could be accessed from a content server,based on a user's profile).

Output from the content analyzer for audio content, may include: beatdetection; musical scale detection; melody extraction; interest pointdetection based on other user data. (e.g. could be accessed from acontent server, based on a user's profile); and voice recognition (e.g.word output, language processing).

The modulation controller (“MC”) or rules engine may include a set ofrules that map conditions, user state (e.g. brain state), andapplication state with a modulation output. A MC takes as input eventsand continuous data and creates signals to control the modulator. A MCis selected based on a presented user interface and user stateestimator. A rule in the MC can take inputs from the user profile (e.g.age), type of content, user state estimate, selections from the UserInterface including Manual Override, Goal, etc. The Output of the MCcontrols the Modulator. A modulation controller may be built to operatewithin a dataflow architecture, so that it can be embedded in the rulesengine by an application or a user.

A non-limiting list of types of rules may include: discrete relations(e.g. decision trees; lookup tables; if/then); controllaws/relationships (e.g. PID controller, threshold based events withhysteresis); method (e.g. reward based neurofeedback paradigms onnormative user data); training systems (e.g. neurofeedback mindfulnessprotocols; state change methods including progressive stimulus withneurofeedback, and iterative relaxation/excitation state hopping); andstimulus based active state estimation (e.g. ERP based cognitive loadmonitoring; and ERSP based attention monitoring).

A modulation controller selector component may be provided, implementedin hardware or software that chooses a modulation controller, or set ofrules, to apply to the content. The modulation controller selector mayinclude a probability model, a local model update function, and a globalmodel update function. The probability model may include a hierarchicalbayes model, that allows for the estimation of what MC should be usedgiven User inputs (e.g. User Goals, Activity, user state estimation,manual override, etc.), application state, and content features. Thelocal model update function may update the probability model based onreal-time user inputs, in particular based on the manual override (thepresentation control command received from at least one user inputdevice). The global model update function may use the data in the userprofile (including recent data produced by the rules engine), and maytypically run in the cloud because of the computational complexity. Ifthe model is updated in the cloud, the updated model may be returned tothe client device along with the local model update function.

A user estimator component may be provided that takes as input acombination of discrete variables and continuous variables. Its outputsmay also be a combination of discrete variables and continuousvariables. Output examples may include: mind wandering, thinking,sleeping, etc., as well as a related intensity.

A user interface (“UI”) may be provided by the computer system of thepresent invention where the user may select or presented with a goal(e.g. to be happy, creative, focused, calm, etc.). The UI may alsoindicate or allow the user to specify a preferred application to achievethe goal (e.g. meditation exercise). The UI may also provide anindication of or allow the user to specify a current activity performedby the user (e.g. doing homework, writing an email, using Facebook,etc.). The UI may also allow the user to self-report activities orevents, and may provide a means for the user to override any applicationof rules or modification of presentation of the digital content throughselection of or manual entry of a presentation control command.

The modulator component may be implemented in hardware or software andmay modulate the digital content by changing audio, video, or text asdirected by the modulation controller. Types of actions that amodulation controller can output to the modulator for implementation mayinclude: proportional (e.g. this is an amplifier that multiplies thesignal by some constant to increase its amplitude or frequency or someother feature; derivative (e.g. the rate of change or acceleration of afeature is used to control the modulator, for instance if heart rateincreased by 20% then send calming audio to user); integral (e.g.accumulation or summing up of features used to control the output, forinstance if a user is in a brain state for 30 seconds then apply areward tone for neurofeedback); state machine (e.g. like a computerprogram); if-then-else (e.g. subset of state machine).

The user profile may be stored on the client computing device or aremote computing device, such as the cloud platform of the presentinvention. The user profile may use demographic and historicalinformation as input and can be updated by a rule executing.

Rules for the rules engine may be pre-programmed in advance to providegeneral rules or a default set of rules for the computer system toapply. Rules may also come from third party developers or shared byusers with other users, possibly over social media that share similargoals, interests, or demographics. Rules can be automatically discoveredand updated. Rules may also come from experience a user has gained usingother applications. For example, in a meditation application, the goalmay be to achieve a quiet mind and focus. This can be transferred torules that are for training and mood management to decrease distraction,increase focus. Also these rules can be transferred to social engagementto prepare for a social engagement, or to health management fordiagnosis and treatment.

Possible functional components of the rules engine may include: state(update) transition matrix; context estimation; prediction; andstimulation. Other functions of the rules engine may include: needs tobe able to generate decisions stochastically based on probabilisticmodels; needs to be able to update probabilistic models (e.g. cloudglobal search and annealing, using a user profile or multi-userprofile); needs to be able to synchronizes sources of input so thatrelations can be established; needs to be aware of time delays ifinteractions between users are needed); needs to have a hierarchy ofrules allow its function to be comprehended by the user; needs toestimate the impact of its actions on the user relative to its goals;provides a scripting language to build interaction with pre-compiledmodules (e.g. scripted rules can run in the cloud and be optimized as inthe case of PYTHONISTA™, or LUA such that the application is sandboxedand can't download malicious code. Rules modules may be downloadable,and may be implemented for example as JAVASCRIPT. The rules engine maybe enable the insertion and removal of new rules; and real-timealteration of a pipeline of rules.

Non-limiting examples of features of rules may include: if visible touser; if created by user; if selectable by user; if detectable by user;if overridable by user; output characteristic (e.g. step response,etc.); input variables; output variables; output certainty;priority/precedence; timescale; and compute-scale.

Rules may be put together in a pipeline like a data flow processor (e.g.PD or max/msp). The rules may be applied to input streams sampled at aregular rate. The rules may be event driven where all required dataneeds to be available at an input before processing. The MC maydetermine a rule to be applied based on a final value received, or maytake the previous values as input, wherein the rules is generated baseon a message or subset of a message.

Different types of rules engines may be used, or differentimplementations of the rules engine may be implement or generatedifferent types of rules. For example, in one implementation the rulesengine includes inference rules (e.g. IF some-condition THENexecute-action). In another implementation, the rules engine includesreactive event condition actions, wherein reactive rules direct therules engine to detect and react to incoming events and process eventpatterns (e.g. the event part specifies the signal that triggers theinvocation of the rule). The rules provided may also implement logic.For example, a rule implemented to the rules engine may include acondition portion that includes for example a logical test that, ifsatisfied or evaluated to be true, causes an action to be carried out.Or another example of a rule ma include an action portion that triggersupdates or other operations on local data. In another example, a ruleimplemented to the rules engine may include backward chaining startingwith a list of goals (or a hypothesis) and works backwards from theconsequent to the antecedent to see if there is data available that willsupport any of these consequents. In yet another example, a ruleimplemented to the rules engine includes opportunistic reasoning whereknowledge elements may be applied either forward or backward, at the“most opportune time”. In a still other example, a rule implemented tothe rules engine includes deterministic rules similar thereby defining adeterministic rules engine.

Example events and conditions may include for example: (i) goals; (ii)events (such as “drowsy”; (iii) current activity (e.g. reading); (iv)time of day; (v) history (such as past responses to the same condition).These events an conditions may be used by the rules engine to (A)determine one or more relevant rules, and (B) apply the rules byoperation of the rules engine so as to establish for example particularparameters for modulating content.

Further possible details regarding a manual override component are nowprovided. The manual override component may be implemented in softwareor hardware. The manual override component may receive or process apresentation control command received from a user. The manual overridemay provide for fine tuning for more or less of the modulation. Themanual override may be applied before the modulation controller hasdetermined any rules to apply, in order to modify selection orapplication of the rules. The manual override may also be applied afterthe modulation controller has determined the rules to apply, and afterthe modulator has applied the modulation to the presentation of thedigital content, in order to cancel or confirm the modulation, or tomodify the presentation further. The manual override may provide for:fine tuning for more or less of the modulation; adjusting the goal (e.g.rules engine behaves differently depending on the goal that the user hasselected); adjustment of the modulation, including immediate adjustment;update or correction of the user's state; customizing the rules to bespecific to the user; helping the system of the present invention toimprove a prediction model by providing better labelled data as reportedby the user; system error identification and correction; and change inmode of operation of the system based on user selection.

Accordingly, digital content presented to the user may be modulated (forexample by modifying the presentation of the digital content) inreal-time both when created by the user or when consumed by the user.

Features extracted from EEG signals vary widely across individuals.Factors such as age, work history, school performance at grade level,IQ, history of drug or alcohol abuse, history of psychiatric orcognitive problems, history of head injury or seizures, current CNSactive medications, medical or neurological disorders, gender,ethnicity, weight and height, cigarette smoker or non-smoker, etc.contribute to this wide diversity. In addition to variance acrossindividuals there is variance of EEG feature measurements within anindividual caused by: drowsiness, agitation, time of last meal, time ofday, etc. (see ref.: L. S. Prichep. Use of normative databases andstatistical methods in demonstrating clinical utility of QEEG:Importance and cautions. Clinical EEG and Neuroscience 36(2), pp. 82-7.2005. Available:http://search.proquest.com.proxy1.lib.uwo.ca/docview/206350047?accountid=15115).

Analyzing EEG signals to make accurate inferences about emotion, mood,and other brain states requires accumulating large annotated databasesboth across individuals and across different factors within anindividual. Moreover, the ability to discriminate across subtle degreesof brain state requires an even broader collection of annotated EEGdata.

The applications referred to in this disclosure that enhance contentbased on brain state, for the most part, may require an externaldatabase of annotated EEG data. As an example, the emotional response ofa new user to a video can be estimated based on the EEG patterns thatother users exhibited where the data and models of the other users arestored in a database (see refs: S. Koelstra, C. Muhl, M. Soleymani,Jong-Seok Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt and I.Patras. DEAP: A database for emotion analysis; using physiologicalsignals. IEEE Trans. Affective Comput. 3(1), pp. 18-31. 2012. Available:http://resolver.scholarsportal.info.proxy1.lib.uwo.ca/resolve/19493045/v03i0001/18_dadfeaps.DOI: 10.1109/T-AFFC.2011.15; and M. Soleymani, J. Lichtenauer, T. Punand M. Pantic. A multimodal database for affect recognition and implicittagging. IEEE Trans. Affective Comput. 3(1), pp. 42-55. 2012. Available:http://resolver.scholarsportal.info.proxytlib.uwo.ca/resolve/19493045/v03i0001/42_amdfarait.DOI: 10.1109/T-AFFC.2011.25).

Emotagging Content Creation

In accordance with aspects of the present invention, an artist orcreator of content may enhance the created content or artwork by addingelements driven by the creator's emotional state.

As shown in FIG. 40, brain state data that is captured during thecreation of a text document may be displayed together with the document.In particular, the brain state may be time coded to be associated withcreation of particular passages of the document. The brain state datamay then be presented in a graph, chart, or by any other indication typeto a viewer of the document together with the document, as shown in FIG.40. In this example, the brain state data is shown in the form of a barin a margin of the document text. The bar shows the emotion of theauthor at the time of writing the particular text located on the sameplane as the emotion indicated in the bar. The viewer may position amouse pointed over particular locations of the bar chart to show moredetail about the text and emotion.

As long as styles that are being modulated by brain state are fairlyambient and passive, emotype could represent a new channel ofcommunication online. If e-mails, text messages, social media message,or any other online text communication were coloured based on emotionalstate, users could gain added insight on the brain state of theircommunication partners. Such a paradigm would depend on thesemodulations being easy to ignore, but easy to read when curious. Forinstance, if the arousal dimension of emotion were paired to blue andred, it would be an easy colour change to ignore, but if one chose tofocus on it, could get an idea of the typist's brain state whenexpressing certain thoughts and ideas.

In the context of a drawing program, brainwave/emotion/state informationmay be introduced by creating a filters or patch for the respectivedrawing program (e.g. Adobe Photoshop). For example, the user's brainstate may be processed, either destructively or non-destructively, inaccordance with the present invention to change presentation of a scene.For example, as the user focused, the contrast of a scene couldincrease. If the user was frustrated, the image could have a redtransparent filter on it with increasing opacity as the user became morefrustrated, or a “shattered” effect could be added on top of the image.The line styles could also change from precise and fine to jagged andthick based on brain state. Similar applications may also apply to movieediting or viewing programs, or to any other program type.

Text-Based Implementation

In one example embodiment of the invention, the brain state of a user isused by the system to modulate the appearance of text input by a user,for example by the user typing text. This may be done continuously andin real time or near real time so as to provide embellishment of thetext without adding workload. The modulation may be implemented throughan adaptive user interface.

Emotional state classification, based for example on applicable displayrules, permits the addition of emotion embellishment similar to the useof emotions, but in an automated fashion, and way that is more closelyaligned with various text portions so as to reflect the variation of forexample emotional state during writing.

The text may be generated by typing or digital capture of handwriting.

In one implementation, the computer system of the present invention usesa plurality of signal ranges to define emotional state classifications.For example a two-dimensional signal may be used such as brain metricsof “relax” and “focus”, and one or more display rules are associatedwith each of these metrics such as for example a particular font face.

Dimensional scaling may also be used. For example, a higher relaxt=larger scale and lower relax=small scale. The same may be true forfocus. This scaling may also be adjusted for aesthetic tasks. Theadaptive user interface of the present invention may also be programmedto provide smooth front transitions as brain state changes, and to varysize depending on brain stage ranges within an emotional stateclassification for example for example by changing serif sizedynamically.

There are many implications of the present technology. For example, areader can consume content in a way that reflects the focus of thewriter on selected portions, which may have bearing on the relativeimportance of particular passages for example. For example, attentionmay be focused on written passages that really matter (as identified bythe technology of the present invention) while giving for examplerelative less attention to detail in other passages.

In one implication of the present invention, written communication ismade more similar to verbal communication in that the embellishmentsserve to allow important content to be emphasized, and also permit lessformality for less important content.

Creation of a verbal or written communication may have different stages,and encoding the communication based on brain state may be moreevocative of the creator's intent depending on the stage. For example,in connection with modulating a written communication with the creator'semotions, an phase during which the communication is being edited forspelling may not be associated with emotions or strong emotions. Incontrast, when a written communication is initially composed, perhaps infree form, and then once editing of the written communication iscompleted and the creator chooses to review the communication again, andexperience the underlying emotions again, there may be stronger emotionsthat can be captured and encoded to the content, thereby evoking moreaccurately the creator's intent or the emotional context of the creationof the communication.

Reading-Based Implementation

The present invention may also be used for reading. For example, thepresent invention may be used in connection with an e-reader. Thee-reader may include for example an e-reader computer program that ismodified to include programming that, when executed, provides thefunctionality described.

In one implementation, when a user scrolls over the text as they read,their bio-state affects the font (font-face, size, color) and thusencodes their reaction to the text as the user reads.

This encoding of reaction based on brain state may be used for a varietyof purposes. For example, the present invention may be used as ahighlighter to embellish content automatically, for example text, toallow the reader to recall it more easily; to draw their attention insubsequent reads; to help them keep focused in the writing; remindingthem of their state of mind/body and its relation to the content.

The encoding may be used to embellish activity feeds published bylinking an e-reader for example to a social networking platform. When auser is reading a particular e-book, this information may beautomatically shared with the user's social network, and the informationmay be encoded with state of mind/body experienced by the user whenreading the e-book.

This encoding is also very useful to the writer (or the writer'spublisher), and to other readers (in a multi user scenario). In thefirst case, the writer may get immediate and detailed feedback about howthe writings affect the reader. In the latter case, many readers maycontribute to the mark-up of a text, making it visible how the writingaffected others as well as forming an automated way to aggregate theresponses of many into a simplified model of how it affects people onaverage. Aggregation may involve generation of statistics regardingengagement of users with particular content, for example using a median,or mean, or mean with outlier rejection using a robust method such asm-estimator).

The information generated may allow writing to be optimized forcomprehension through rewrites or similarly to maximize the readingefficiency through optimized highlighting. It also provides a usefultool for insight and exploration into reader interest.

The computer system may use this information for example to aggregateinformation regarding engagement by users with selected content. Thisinformation may be used in guiding the creation of additional contentfor example in conjunction with an e-learning platform. Also, a socialmedia platform may match users that react to content in similar ways.Other uses and applications may be possible that leverage the advantagesof the present invention.

In one particular implementation, the communication utility may be ajournaling system. The enhancements of the present invention makesjournaling more engaging, and stimulates insight and in some cases morehonest an insightful communications. Particularly in the context of theuse of journaling as a method for therapy, embellished journal contentcan provide useful content into non-verbalized emotions.

FIG. 3 for example illustrates how changes in the state of concentrationmay be captured and shown in regards to a user who is performing a taskor completing an exercise.

FIG. 4 presents a representative output of the user interface, in thiscase display output associated with a self assessment from a journalingapplication, where content captured is enhanced with a modulated fontand word cloud. This type of display provokes thought in users.

The user interface for displaying the enhanced content may also beaccompanied with other associated information such as a mood chart whichmay be used to show an accurate representation of the mental state forthe user.

Modulation of Social Media Content and Social Media Interactions

Emotions can be added via brainwaves to one's content feed. For example,on Facebook™, one might post that they are sad. They can do this bysharing their brain state via brainwaves as an emoticon or icon next totheir text post. For example, the text post “i am feeling sad today”,could be accompanied by a sadness icon or emoticon generated as a resultof the users felt sad state. It could also be expressed in the text ofthe posting itself (in font), in the colour of the post, or some otherindication on the user's profile photo or status update such as colour,text, video, etc, that would indicate state.

In a further implementation, others in the users social network couldthen post their own feelings (sad, happy, caring, loving, positive,etc), to show support for the users emotional state, whether empathetic(e.g. also sad, or angry the person is sad), or supportive by sharingtheir, for example, happy thoughts.

These other's supportive states and feelings can be expressed on thepage by similar means (e.g. an emoticon next to a profile photo or post,in the profile, through colour, etc.).

There is also the possibility for the user to have an arena in which tocapture and display the emotion that has been shared with them. So if afriend shares positive feelings with the user, that could show up withinthe social network, as a graph, series of emoticons, in profile picture,as colour, etc. In this way, users could share their “good vibrations”or “positive thoughts” with the user, and the user could capture them.The user could also receive a notification via email of the positivethoughts they have received.

In all these examples, heart rate and other bio sensors could also addsignal or valance information. In particular, a measure of heart ratemay determine a level of user arousal, particularly in conjunction witha determined brain state.

Facebook Like Via EEG

Like and dislike can be determined via EEG. One method of doing this isvia hemispheric specialisation. There may be greater frontal right alphaactivation relative to left activation when one is presented (real orimaginary) with something one likes. Similarly there is greater leftfrontal alpha activation when one dislikes something.

A “like” or other status in a social network can be determined via EEG,and shared in the social network by tagging content, photos, newsstories, news feeds, status updates, etc via brainwave.

HRV, EEG and other bio measures can also add levels of arousal, andadditional cross-measure for the degree of like.

For examples, items that are “liked” can be voted up, and items“disliked” can be voted down, or presented less frequently.

Tagging Items You “Notice” in Social Network

Whereas in Facebook we have a “like” button that one pushes manually ifone likes something,

With EEG one can immediately tag stimuli that one notices. When onenotices something salient in the environment, a p300 brain wave mayfire. This can be detected via EEG, and the item that was salient linkedto the time course of the p300 firing. Items that were salient ornoticed can then be tagged in a social network and shared, or if theyare items already within the social network, they can be tagged withthis “noticed” notification. This can cause them to automatically enteryour social network (for example migrating from News Feed to yourtimeline), tagging or book marking them, sharing with others that youlike it, or influencing the popularity and frequency of presentation ofthe content (influencing the Facebook social graphing back end). Thisnotice allows you to curate your content based on what has caught yourinterest or attention.

The present invention enables new methods for tagging online contentusing emotions or brain state. This type of tagging provides a newmechanism for adding authenticity to online communications, such ascommunications via social networking sites for example by automaticallytagging content based on emotional state or brain state, using thepresent system. This can improve the proximity or authenticity ofconnections between for example social network users. The tagging ofcontent with emotional or brain state may also be used to filter socialnetworking content, such as activity feeds.

An online identity is an actively constructed social representation ofoneself on the Internet. Digital curation may be performed by anindividual by hand selecting what the individual believes is greatcontent, commenting and or providing context with a unique perspectiveto accompany a piece of content. Current online identities allow for thecuration of one's self-presentation driven by a user's subjectiveopinion of content. Information, however, that is extracted from auser's brainwaves may add natural authenticity to an annotation ofposted content which is more genuine than voluntary approaches tomanaging online identify. A user can be more self aware of the user'sbrain state as the user creates and/or curates content and can be usedto determine how one's mood is influencing one's content creation.

Typical online communication tends to be ambiguous because it subtractsmany of the normal cues of face to face communication (e.g. bodylanguage, facial expression, etc.). Brainwaves provide an opportunityand medium to add an emotional tone which will help users interpret thecontent created by others. The feeling of social intimacy can beincreased when there is a more genuine expression of emotion. Contentthat is a more genuine representation of oneself has a greater chance ofinspiring others to talk back, thereby enhancing engagement through theapplicable social networking platform for example.

Technology has contributed to greater social isolation of people and thesharing of emotional state can help partially mitigate isolation.Emotional sharing is often encouraged in a number of group situations.The examples of groups interested in emotional sharing include:families, support groups, or any kind of tight-knit group. In oneimplementation of the computer system of the present invention,emotional sharing can occur automatically based on encoding ofcommunications.

In a non-limiting embodiment of the present invention, the computersystem may tag content based on a user's brain state or determinedemotion(s). In one example, a user may post content to a social mediaservice such as Facebook™. The user's post may include text, image(s),link(s), and/or video(s) and may be further annotated or annotatablewith a summary of the user's brain state. Friends of the user on thesocial media service could, for example, click to see details of braindata of the user recorded while the content was created by the user. Anyuser may intentionally or passively tag a photo, image, or other contentwith emotions or other state information. When another viewer sees thecontent, it may be possible to view the state information of a personwho previously viewed the content. For example, a user may see asibling's wedding photo and exhibit a happy brain state, which iscaptured by the computer system of the present invention. The computersystem may then tag the photo with this determined brain state, and thephoto may be shared with other users together with the tagged brainstate.

As mentioned earlier, the present invention may be used to filter socialmedia interactions. Many social media websites (e.g. Facebook) have manyusers that curate “fake” personas. In one example of implementation ofthe present invention, the computer system of the present invention tagsa person's post with their emotional state to show the world that theyare being genuine. These posts benefit of increasing user's emotionalself-awareness and fostering genuine communication and strengtheningrelationships. The user may want to post something genuine and writes adraft text of the post. A semantic analyzer may be used to determine theemotional tone of the post. The user's brainwaves may be analyzed andcompared to the text. If there is not a large gap between the semanticanalyzer and the emotional state as estimated by analyzing their brainwaves then a genuine tag can be added to the post or one-to-onecommunication. The rule may be defined as follows: IF (User=Brain stateof Mindfulness is Moderately balanced) THEN ACTION: “USER=ModerateMindfulness Expert”.

Optionally, any brain state information tagged to a photo or othercontent, may either identify the user who exhibited the brain state, orthe brain state information may be anonymized. For example, a tag on thephoto may indicate that “10 people were happy when viewing this photo,and 2 people were unhappy when viewing this photo”, without necessarilyidentifying which user experienced which emotion. An exemplary post withbrain state information is shown in FIG. 20 which shows a social mediaposting by a user indicating brain state data for the user on a givenday. Brain state information may also be tagged to a video posted to asocial media service, or elsewhere.

Further possible details of implementation of the present invention forsocial networking are now provided.

Typically there is a list of people that the user knows and communicateswith on social media, or in various chat services, or othercommunication services, with an indicator of the user's or the user'scontact's current status (online, away, busy, etc.). With the contentmodulation system, people that have emo-state information available, mayshare this information and have it available to enhance the interfaceapplication for the user. This information would be used to modulate theuser's status indication or may be used to modulate the user's currentactivity, or may be presented as content in itself (user state ascontent). The modulated content may presented in the same location (ornext to) where their status is indicated. Different user will havedifferent preferences on the display, and an interface may be providedto allow the user to customize it and also customize what is availableto others. For example, a user will want to limit how much of the user'semo-state is available, and to whom. Different levels of privacy may beassigned to groups or individuals.

For example, the user's current status may appear as a colour that maybe modulated using the user's current level of engagement and the user'smanually selected state.

Where the user is available for chat and is not trying to concentrate(and therefore would generally be welcome to chatting), green wouldindicate that the user's has low mental load. High mental load may berendered as orange. When the user is available for chat, but is tryingto work (so they would be not want to be distracted if they are in agroove): low mental load+unfocused (distracted), indicator would begreen; low mental load+focused, indicator would be orange; and highmental load+focused, indicator would be red.

The current status may be augmented with a rendering of user state thatshows some of their state history. This can help other user's have abetter idea of whether they want to disturb them or help them interprettheir behaviour (humans are still much better than this than computeralgorithms). The user's state history may be rendered as a scrollingline graph showing some of their recent brain state history. Brain-statemay be rendered as a one-dimensional, or multi-dimensional graph. Animplementation may be a one dimensional graph (so that it is not toocomplicated for other users to understand), where the x axis of thegraph is time and the y-axis is related to brain-state, such as mentalload. Greater on the y-axis is related to greater mental load. The timeaxis of the graph may be linear time (time ranges of 5 min to 5 s havebeen found to be particularly useful). Another useful feature is to usea non-linear time-scale (logarithmic typically) where the timescale iscompressed more at the end of the graph representing more in the past.

Another column of information may be integrated into the UI thatdisplays information about the user's real-time brain-state resonance(“in-sync”) with their friends. This type of display is very useful forhelping users know who they may have the best communication with, italso helps create feelings of connection between users and theirfriends, without the need for direct communication. The display may havedifferent privacy settings than other aspects of emo-state modulation.For example, a user may not want to reveal what their current state is,but they may not mind a display of when they are in sync with another.

Useful measures of “in-sync” or “resonance” may involve analysisrelative to a common stimulus. For example the application interface mayhave a dynamic visual aspect (like a video playing in a portion of theinterface). Time synchronized analysis is done between the user'sbio-signals and the features in the video content, yielding measuressuch as mental engagement with features in the video content.Correlation analysis is then performed between the measures fromdifferent users and used to estimate how “in-sync” pairs or groups ofusers are with each other. greater correlation being associated withmore “in-sync”.

The common stimulus may be each user's real-time emo-state, when it isrendered through the application interface. For example, each user'sstatus is modulated with their emo-state (modulation may be the colourand intensity of the user's status indicator). Time locked analysis (forexample correlation analysis between hue and intensity time-series ofanother user's status indicator, and the application user's bio-signals)is conducted to produce measures of coherence between the user and otherusers. User that are considering their list of friends will producehigher coherence between their biosignals and their friends statusindicators, with higher coherence with those friends in the list thatthey are paying particular attention to. The advantage of computingcoherence between the user and the status indicators rather than betweenthe original bio-signals is the reduced amount of information transferthat is necessary, the reduction in privacy concerns. It also involvesconsiderably less computation. Coherence scores are in themselves a timeseries that is shared with the application or website over the internetconnection. The application (website) server can then process thecollection of coherence scores between user's that are visible to eachother, to create measures of “in-sync-ness” or “resonance”. Thisanalysis can be as simple as adding up the coherence scores over theappropriate time frame based on the individual user's behaviour, howeverin practice a probabilistic model is learned that can be user specificto improve the accuracy of the “in-sync”-ness measure. Differentoptimization criteria may be chosen depending on the application. Forexample, in a social site, the model would be optimized to producelonger conversations with more positive content, or greater contentsharing with other users. For a work environment, the model may beoptimized for greater discussion of work related issues. or lessdistraction (by monitoring how quickly people end the conversations).

Users may elect to share an experience together. For example a group maydecide to listen to the same piece of music or video (ideally timesynchronized). Analysis in this case would be similar to the firstexample, however stronger conclusions because user's will be focusingmore intently on the same content. In example the user interface may beexpanded to include a higher fidelity real-time modulation of the user'sstate indicator, since user's will be able to more directly connect withother individuals state indicators (and better understand relationshipsthan a computer algorithm) because of it's time locked nature (timelocked because the user's are all experiencing the same thing at thesame time). In the case where there timing is asynchronous, or whenuser's attention may be moving to different portions of the applicationinterface (or to other things in their environment), a computeralgorithm would typically do better as it can look for correlationsacross large time ranges.

Other useful measures of “in-sync” or “resonance” may also be measuredwithout a shared content experience. In this case a deep user profilemay be useful as it can allow for more robust emo-state estimations andcan be analysed with respect to interaction data with friends to yield aprobabilistic model for what states are ideal for specific users tointeract in. In this way “resonance” may be calculated by usingemo-states from pairs of users (this may be computed on a trustedserver, if people don't want to share their information). A “resonance”estimate would need emo-states of both users and the probabilisticmodel. This model would typically be initialised with a symmetricrelationship (where users of like states yield the highest “resonance”probability) or with a more sophisticated model based on available data,but would be improved as more user data is collected (for example, highresonance may be estimated between a user who is “sad” and one who isfeeling “loving”). Resonance may also be introduced into interface lessdirectly by having user's with greater resonance come up in theinterface as, for example, “maybe you would enjoy talking to friend X”.

Another example is shown in FIG. 21 which shows a video posted to asocial media service. The video includes an indication of the user'sbrain state while watching the video. The user's brain state may havebeen recorded and time-coded, then the computer system may have uploadedthe time-coded brain state information to the social media service totag the video. The tagged brain state information may be shown as anoverlay of the video, as shown in FIG. 21, or may appear in a separatetable or window, or may appear in textual instead of graphical form.

An example of video tagging is shown in FIG. 22 which provides anindication of the user's brain state upon completion of presentation ofthe video. The user may be notified of a particular brain state responsedetermined from the user's bio-signal data obtained by the computersystem of the present invention. The user may then be prompted to sharethe brain state response or not, or to maintain the brain stateinformation, but only show it to the user, and not share it. The optionsfor sharing this information may be shown as buttons to click or checkoff to the right of the video window, as shown in FIG. 22.

While wearing the bio-sensor(s) of the computer system of the presentinvention, the user may watch a video on a video sharing service, suchas Youtube. The computer system may determine that the user isexhibiting a particular brain state or emotional response and notify theuser accordingly. The user may respond to the notification to share theuser's response on Facebook, Youtube, or any other suitable service. Theuser may manually login to the respective service or the computer systemmay be configured to automatically login for the user. The user may beprompted to select users or groups of users with which to share thebrain state data. The user may include text for posting together withthe brain state data, and optionally a link to the video.

In the case of Facebook or any other social network, the user's timelineor equivalent may be shown with a post including the user's brain statedata. Such a post may be referred to as an “emotagged” post. Other userswith whom the post was shared may view the emotagged post in a formatdetermined by the user. For example, the user may choose to present thebrain wave data as a “happy face” representation, as a graph or chart,as text, by a colour or any other graphic or embellishment orenhancement or modification of the shared content. For example, theshared video may be shared with a coloured border surrounding all orpart of the video window. A colour of the border may be selected to beindicative of the brain state or emotion of the user who has shared thevideo, as shown in FIG. 26. For example, a red border may indicate thatthe user was angry when watching the video; a blue border may indicatethat the user was sad; and a green border may indicate that the user wasnot experiencing any significant emotion while watching the video. If auser watches a video while not wearing the bio-signal sensors, the usermay opt to re-watch the video while wearing the bio-signal sensors inorder to record bio-signal data for sharing.

FIG. 23 shows another implementation of the present invention, where aparticular shared video is locked for viewing until the viewer achievesa particular brain state. The uploader or sharer of the video may selectthe brain state, or intensity of brain state to achieve before viewingis granted by the computer system or video sharing service. The videomay be unlocked once the brain state has been achieved a single time, orthe viewer may be required to maintain a particular brain state for theduration of the video in order to continue to watch. Optionally, shouldthe viewer's brain state not meet the established requirements, then thevideo may pause or stop. Optionally, a message may be displayeddirecting the viewer to “focus” or try to achieve the desired brainstate in order to resume playback. Accordingly, a set of rules may beassociated with content that require the user consuming the content toachieve certain brain state parameters in order to be permitted toconsume the content, or unlock particular content sub-components.

FIG. 24 shows an implementation of the present invention, where a sharedvideo is being viewed and the viewer's bio-signal data is being trackedand recorded. As both the video and the bio-signal data is time-coded,an overlay or separate window may be presented on the video showing theviewer's bio-signal data, or brain state response to viewing the videoat the particular time codes indicated by a video playback progressindicator. An icon or other indicator may also be shown either in astatus bar of the video window, or elsewhere, to indicate whethercurrent bio-signal data is being recorded, as such bio-signal datarecording may be toggled by clicking an appropriate control inputdisplayed at or around the video playback window, or by pressing asuitable keyboard key, or by providing a suitable voice control command.

FIG. 25 shows an implementation of the present invention where photos orother content are arranged, organized, filtered, or otherwise groupedand displayed by brain state information tagged thereto. The userposting the content may have tagged each content item with brain stateinformation when uploading, posting, or sharing the respective contentitem. Accordingly, content items tagged with a brain state of “happy”may be retrievable and displayable together by suitable query.

When a user is consuming digital content presented to the user, theuser's current brain state may be communicated to the user. For example,a graphic may appear on a display screen of the computing device used bythe user to indicate the user's current brain state, as shown in FIG.27. This graphic may take the form of an avatar or representation of aperson or face, that is made to appear happy when the user is in a happybrain state, or is made to appear sad when the user in a sad brainstate. A brain state graph may also be shown indicating a value of theuser's present brain state, which may be updated in real-time orpseudo-real-time.

Optionally, when either consuming presented digital content, or viewingshared content having associated brain state data of a user, a graphicalrepresentation of the brain state data available may also be presentedwith radiating lines or rings that increase and decrease in size,intensity, or in any other property based on the brain state data. Forexample, as the brain state data indicates more intense feelings ofhappiness, the radiating lines may increase in intensity, and whenfeelings of happiness decrease, the radiating lines may decrease inintensity.

FIG. 28 shows a non-limiting method of the present invention. The usermay click to share content (e.g. a video) on Facebook at 2800. At 2810,the user may be asked to include emotion or brain state data with theshared content. If the user clicks to share this data, the user may beprompted to put on the bio-signal sensor(s) at 2820. At 2830, the usermay watch the video, and the user's bio-signal data, and determinedbrain state data may be recorded by either the user's local computingdevice, or a remote computing device in communication with the user'scomputing device. The recorded bio-signal data may be time coded tocorrespond to time codes of the content presented. At 2840, the user maybe prompted to review the recorded/determined brain state data. Oncesatisfied, the user may click to submit the shared content with theassociated brain state data. At 2850, the shared content and associateddata may appear on the user's Facebook profile. The user may share suchcontent on other content sharing services as well.

FIG. 29 shows a non-limiting example of a shared video in accordancewith the present invention. Other users may have subsequently viewed theshared video, also when wearing their respective bio-signal sensor(s).The brain state data of the respective users viewing the video may alsobe shared and associated with the shared video. Accordingly, this brainstate data may also be displayed to subsequent users accessing orviewing the shared video. The brain state data may be shown in atime-coded graph, as shown in FIG. 29, or by any other format.

FIG. 30 shows a non-limiting example of a representation of a usercommenting on a post on Facebook. Together with the user's textcomments, if the user's bio-signal data is also being recorded, agraphical indication approximating the user's brain state may beassociated with the user's comment to indicate. Optionally, thisgraphical indication may indicate whether the user was happy or sad whenposting the comment, and may optionally indicate an approximateintensity of the determined brain state.

FIG. 31 shows a non-limiting example of a method of the presentinvention. The user may be viewing a video on Facebook, Youtube, or anyother video service, or simply on the user's computer in a video playerapplication, while wearing the bio-signal sensor(s). At 3100, An icon orother graphical or audio indication may be provided indicating to theuser that a particular brain state has been detected from the user. Forexample, the icon may indicate that the user is experiencingabove-average happiness. At 3110, the user may click the icon, orinteract by any other means, to share the video. At 3120, the user maybe prompted to provide a text comment for the shared video or sharedlink to the video. The user may be prompted to share the determinedbrain state, either by sharing an approximate emotion associated withthe entire video (e.g. happiness) or by sharing more detailed time-codedbrain state data of the user. At 3130, the shared video is posted to thevideo sharing service with the user's comments and optionally with theshared emotion or brain state data. An embedded video player (such as aYoutube video player) may be implemented in an application, or webbrowser using an Application Programming Interface (API) supplied byYoutube. Other video players may provide similar APIs. Once a videoplayer is embedded into an app, it may be polled to determineinformation, including: YouTube Video ID specifies and identifies theplayed video; video player status (e.g. not started, ended, playing,paused, buffering, video cued; elapsed time in seconds since the videostarted playing; and total bytes of video that can be used to deriveduration of the video.

Tagging videos or other content with brain state data, or “emotagging”,may be accomplished by installing an application on a user's computingdevice. The application may include an embedded video player or the appmay communicate with a web browser application installed on the user'scomputing device. The browser may provide the ability to query the videoplayer for information and send the information to the app (or the appcan query the embedded player and receive the information), theinformation including: video identifier (“ID”) (may specify and identifythe played video); video player status (e.g. not started, ended,playing, paused, buffering, video cued; elapsed time (in seconds sincethe video started playing); and total bytes of video that can be used toderive duration of the video. The user may open the browser and visit avideo service such as YouTube while wearing the bio-signal sensors andsearch for a video. The user may click a video from the search results.The app, or a separate app interfacing with the bio-signal sensor(s) mayreceive a message from the player that the user is watching a videoidentified by the video ID. The app may assign a session ID based on thecurrent calendar date and time. The app may also assign informationincluding: User ID; Activity Type (e.g. “watching video”); and app dataincluding Signal Type (e.g. EEG), timestamps per EEG sample, timestampsassociated with start and stop of watching the video, and raw EEG dataper channel. The app may label the timestamp “start of video” into theEEG data. The app may poll the video player to get elapsed time ofvideo. The elapsed time plus the start time may be used to labeltimestamps in the EEG signal. Event timestamps of the location in videomay be sent if video is paused. There should be an association of thetimestamps of the EEG recording with the progress timestamps of thevideo in order to associate events in the video with particularbio-signal data at particular times. This may be accomplished by usingevent timestamps that have the timestamp as well as an event label suchas paused video. The app or the cloud platform may analyze the EEGsignal for classification into an emotion somewhere in the continuumfrom sad, stressed to happy, elated corresponding to time scale of thevideo. A graph of this continuum may be displayed as well as the averageand peak emotions felt while watching the video. The user may click ashare button located at, under, or near the video. Clicking a Facebookicon in the sharing options pane that appears may prompt the video to beshared on Facebook. The user may be prompted to login. The user may thenbe prompted to select which users may see the video (e.g. Friends Only,Public, Family, etc.). The user may also be prompted to determine howmuch, if any of the emotagged data is to be shared (e.g. full continuumof emotion along timescale or just one word summarizing the user'semotion). The user may be prompted to enter a message to post with thevideo. Clicking a share link button may then post the video with theoptionally attached emotagged data. The EEG signal and analysis may beprovided to the user's profile in the cloud platform.

In a non-limiting exemplary embodiment of the present invention, one ormore digital content items may be presented to the user, in succession.The user's bio-signal data may be measured by the bio-signal sensors andthe user's brain state may be determined. For each content itempresented, the user's bio-signal data or brain state data may beassociated with the respectively presented content item. This feature ofemotagging content items in succession may be referred to as“emocrawler”. This may be accomplished by tagging the content item withmetadata, or storing a separate data store with the associated data, orby any other means of association. The user may cycle through thepresented content by clicking buttons such as “previous” or “next” orany other navigational buttons. The user may be asked to confirm thedetermined brain state determined by the computer system beforeassociating it with the presented content item. Content items may thenbe later retrieved, grouped, displayed, presented, filtered, orprioritized by the respectively associated bio-signal data or brainstate data. In particular, the associated data may be used to curatecategorize and/or curate the user's content based on the user'semotional reaction to the content. For example, a video montage can beedited based on the editor's emotional reaction to the video content. Asshown in FIG. 32, a brain state meter may be displayed when presentedeach content item. The brain state meter may update in real-time orpseudo-real-time while the user views or otherwise consumes the content,or the meter may display a previously recorded brain state associatedwith the content. The user may be asked to re-record new bio-signal datafor the content, especially if the user wishes to discard the currentlyassociated data. FIG. 33 shows sorting a list of content items by brainstate data (e.g. emotional valence). The content items may be sorted byhappiest associated brain state or saddest associated brain state, or byany other associated brain state. Optionally, when the user isnavigating through a plurality of content items (e.g. vacation photos)and associating a brain state with each photo upon viewing the photo,the photos may then be sorted by emotional response. A list of all theemotional responses associated with all the photos in the photo set(e.g. all the photos grouped for a particular event or vacation, etc.)may also be shown along a side of the presented content. Clicking on oneof the associated emotions may cause a specific action to occur on therespectively tagged photos. For example, all photos tagged with thatemotion to be displayed in thumbnail form, or a slideshow may begin forthose photos.

Tagging content items in this way may be advantageous due to theproliferation of social media which encourages users to generate andshare an abundance of content, optionally with other users over theInternet, which directly relates to the user. This abundance tends tohave a very low signal to noise ratio, as posts, pictures, events,videos, etc. are produced and shared every day. Brainwaves represent aquick and entertaining way to genuinely annotate content and increasethe signal to noise ratio on any user's massive online store ofself-referential content. Just like Facebook is a cloud-based databasefor photos and life events, the present invention may provide for acloud-based repository for brainwave data and processed brain data (e.g.with emotional state or level of focus, etc., determined), that can bestored separately or together with Facebook's content. The user may alsotag content on other social media sites and through other socialmediums, such as Twitter, My Space, Snap Chat, BBM, SMS, etc.

In another non-limiting application of the present invention, journalcontent may also be emotagged. For example, a user may write a diary orjournal for therapeutic purposes. The user's emotion(s) may be detectedand recorded using the computer system of the present invention. Theseemotions may be used to tag passages of the journal with emotions thatthe user may not even be aware of. An avatar may be used to representthe emotions or mood felt by the user making a journal entry. The avatarmay be animated by emotions that are driven by the actual emotions ofthe user as detected by the computer system of the present invention. Bybecoming more aware of the user's emotions tied to certain journalentries, the user may gain increased self-awareness which can betherapeutic. Content could be shared with a spouse or doctor. Text tospeech conversion could read journal entries back to the user modulatedwith the emotion that the user felt when writing the respective entry.The journal may be written in a word processing application with timestamps synchronized to time stamps of the recorded bio-signal data bythe computer system in order to determine what journal entry triggeredwhat brain state in the user. Writing therapy may have a positive impacton health outcomes such as a reduced number of physician visits,improved immunity, and a reduction of negative emotions (see reference:Pennebaker, James W. 2003. Writing about emotional experiences as atherapeutic process. p 362-368 Psychology Press, New York, N.Y.). Inaddition, increasing self-awareness of unconscious motives, and beliefsmay beneficial to mental health (see reference: Wilson, Timothy D., andElizabeth W. Dunn. 2004. Self-knowledge: Its limits, value and potentialfor improvement. Annual Review of Psychology 55: 493-518). Similarly,other digital content including text messages, emails, word processordocuments, presentation documents, or any other type of digital contentmay be tagged with brain state data, more general bio-signal data, oremotions either generally or time-coded with creation or viewing of therespective content. In a non-limitation example, the user may compose atext message, wait for a period to settle on particular feelingsregarding the text message, and the text message may be modulated basedon the user's determined brain state. The user may encode a dynamicfeeling such as a 30 second movement of emotion. The user couldre-examine the user's emotion, adapt, iterate, and determine if it isconsistent with the intended effect. This may provide for acollaborative creative process that is done with the computer thatcycles through compose, feel, compose, feel, etc.

FIG. 34 shows another non-limiting embodiment of the present invention.A video chat window is shown, showing the other chat participant wearingbio-signal sensors of the computer system of the present invention. Inthis example, the video chat is specific to a technical support videochat where the customer, the chat participant shown is determined to befrustrated by the computer system of the present invention. The chatparticipant's brain state data may be communicated to the other chatparticipant (e.g. to the technical support representative) who may beprompted to offer a discount, coupon, or other incentive to the otherchat participant. Accordingly, a user's brain state may be communicatedto other chat participants in video or audio chat conferences. Therespective user's brain states may be shown to the user who generatedthe brain state, and also to any other users participating in the chat.The presented brain state may be representative of the respective user'scurrent brain state, or a full or brief history of the user's brainstate may be shown or available to be shown, from either a part of orfrom the beginning of the communication session.

FIG. 35 shows another non-limiting example, where the received brainstate data of a chat participant, shown on the leftmost box of FIG. 35,may be used to overlay a visual representation of the emotion over thevideo chat image of the chat participant experiencing the particularbrain state. Examples of such overlays are shown in the remaining boxesin FIG. 35. For example, if a chat participant is determined to beexperiencing a happy brain state, the video stream of the user may bemodified to overlay happiness graphics, such as a halo, sunshine rays,or other graphics, over the video image. The contrast, brightness, orother properties of the video stream may also be modified accordingly.Where the user is determined to be sad or angry, other graphics, such aslightning bolts, horns, or angry eyebrows, or darker colours may beoverlayed on the video image, or other video properties may also bemodified. When the user is disinterested or confused, or judgmental, theuser's brow may appear furrowed. When the user is sad, tears may beoverlayed on the user's face. The video image itself may also bedistorted to enlarge certain aspects of the chat participants visualappearance, or minimize other aspects, according to the chatparticipant's determined brain state and any modification rules providedby the computer system of the present invention.

FIG. 44 shows another non-limiting example where a list of a pluralityof callers are waiting in a queue. The length of time each caller haswaited for is indicated, as is the respective caller's brain state.Calls may be answered in priority of the most frustrated or angry callerto the least frustrated or angry, or by any other brain state measure.

Other examples of brain states affecting live or asynchronouscommunication between people may include: emotions change robotictelepresence (e.g. Robotic flower reflects communication partner's brainstate. Robotic telepresence body language reflects communicationpartner's brain state. Emotional state of one partner can be sonifiedand sent to the other partner. The distraction level of thecommunication partner can be monitored, using for example the computersystem of present invention for measuring brain state and phone inaccordance with the present invention can give feedback to thecommunication partner for example on the other party's distractionlevel.

In one possible implementation, the present invention can provide anenhanced communication tool for individuals with cognitive deficits (forexample people that have difficulty expressing themselves such as peopleon the autism spectrum or having dementia). Such individuals can usedthe system of the present invention to represent their emotions topeople around them. This would make it easier to understand a personwith a cognitive deficit. A caregiver can more easily recognize when theperson being cared for is upset or agitated but may not have externalsigns of agitation.

Another non-limiting embodiment of the present invention may be providedwhere, as previously described, content viewed or created by the usermay be tagged, sorted, or ranked based on respectively associated brainstates. The computer system of the present invention may use thisassociated information in order to rank user preferences or providerecommendations to the user. For example, the user may effectively rankpreferences of music, video, movies, television shows, or other contentby the computer system processing the user's brain state associated witheach of the respective content items. The brain state information may beprovided together with the user's personal content preferences.Preferences provided by the user may also be used by the computer systemto fine tune the brain state determination. This may allow for increasedaccuracy of content recommendations to what the user would actuallyprefer. For example, particular content items (e.g. movies) that havebeen viewed by the user and have respective user brain state dataassociated therewith may be presented to the user in a list. Therespective associated brain state data for each content item may beshown. Content items determined to produce a happy, alert, aroused orother “positive” brain state response may be determined to be preferred.Other content items or movies having similar properties to thosedetermined to produce the positive responses may be recommended to theuser. Optionally, other users that have responded similarly to themovies viewed by the present user, may have experienced positive brainstate responses for other movies as well. Those other movies may then berecommended to the present user. Optionally, other users havingbio-signal classification profiles, or user profiles, similar to thepresent user, may be identified by the computer system of the presentinvention and content items preferred by those respective users may alsobe recommended to the present user.

Rating content is generally done in prior art systems and solutions in asubjective way. In contrast, brain state can add an objective element torating content. Typical content rating systems involve a single rating(e.g. number of stars from 1 to 5) which takes place after a userengaged with that content. Live brainwave algorithms could get morenuanced detail subconsciously while users are engaging with the content.Further, the temporal nature of a real time measure would allow therecommendation engine to detect a user's response to a specific moment,or specific sections of the content. Machine learning algorithms whichcurrently underlie most recommendation systems may benefit from havingrecommended content also measured based on output from the computersystem of the present invention. in terms of brain state algorithms ofthe present invention. Machine Learning algorithms typically use atraining paradigm, thus real-time and immediate feedback on the accuracyof every recommendation may be of benefit to the improvement ofrecommendation engines in general.

Shown in FIG. 13 is a non-limiting implementation of the presentinvention showing a client computing device 14 in communication with abrainwave cloud platform 10 of the present invention. The clientcomputing device may include a bio-signal processing application, asdescribed, linked through a web browser extension or video playerextension to a web browser or video player. The web browser and/or videoplayer may connect to a video sharing web site which includes a contentrecommendation engine. Optionally, as shown in FIG. 14, the bio-signalprocessing application may interface directly with the web browser orvideo player. The application supporting emotagging may be installed onthe user's computing device. The application either may have an embeddedvideo player or may communicate with the user's web browser. The webbrowser may have the ability to query the video player for informationand send the information to the application (or the application canquery the video player through a browser extension to receive theinformation), the information including: Video ID specifies andidentifies the played video; Video player status (not started, ended,playing, paused, buffering, video cued); Elapsed time in seconds sincethe video started playing; and total bytes of video that can be used toderive duration of the video. The user may open the browser and send aHTTP query to a web site that hosts the video and logs in. The web sitemay send an HTML page to the user's browser with a search box for videosplus recommendations for movies to watch based on the user's preferencesas stored in the web site's “Content Recommendation Engine”. While theuser is wearing the bio-signal sensors (e.g. an EEG headset), the usermay enter keyword text to search for a video. This HTTP message is sentto the web site. The web site returns HTML with a list of videos thatmatch the user's request possibly sorted by a ranking algorithm weightedby the user's preferences. The user may click on a video from the searchresults. The application may receive a message from the video playerthat the user is watching a video identified by Video ID. Theapplication may assign a session ID based on calendar date and time. Theapplication may also assign information including: User ID; ActivityType “watching video”; and application Data including Signal Type (e.g.EEG), timestamps per EEG sample, timestamps associated with start andstop of watching the video, and raw EEG data per channel. Theapplication may label the timestamp “start of video” into the EEG data.The application may poll the video player to get elapsed time of video.The elapsed time plus the start time is used to label timestamps in theEEG signal. Event timestamps of the location in video are sent if videois paused. There needs to be an association of the timestamps of the EEGrecording with the progress timestamps of the video if we are toassociate events in the video. These are done by using Event timestampsthat have the timestamp as well as an event label such as paused video.The application may send raw EEG data plus timestamp data to theBrainwave Cloud Platform of the present invention. The platform mayanalyze the EEG signal and classifies into an emotion somewhere in thecontinuum from sad, stressed to happy, elated corresponding to timescale of the video. A graph of this continuum can be displayed as wellas the average and peak emotions felt while watching the video. Theplatform may send the emotional classification to the user'sapplication. The application may associate the Video ID with theemotional classification received. The application may send this to theBrowser. The Browser may send a message with the Video ID and emotionalclassification to the video web site. The video web site may useinformation about the movie to build a feature set for that Video ID.The feature set can include emotional classification based on the user'sbrainwaves while watching the video. The video web site may build aclassification model and or rules using machine learning. The web sitecontent recommendation engine can make recommendations based on therules it has learned about the user's preferences.

In a non-limiting example, educational or learning content may beprovided by the computer system of the present invention where thepresentation of the content is modified based on the level ofconcentration or focus or other brain state experienced by the user,determined as described. Similarly, in another example, a feed ofweb-based content (e.g. a live or pre-recorded feed) may be changedbased on user's brain state. The delivered content can be made moreengaging, relaxing, etc. based on the brain state of the user. Similarlya scroll rate of a text or web browser view can be modified based on adetermined level of focus/alertness of the user which may be indicativeof the user's ability to retain information. The difficulty level of thelearning content may be modified based on the degree of focus of theuser at the relevant time.

FIG. 36 shows a non-limiting implementation of the present invention,where a user's brain state response determined in accordance with thepresent invention may be compared against the respective brain stateresponses of one or more other users. As shown in FIG. 36, s first userand a second user may have each watched the same content items, ormovies, while having their respective brain states determined andtime-coded corresponding to the timing of the respective movie. Eachuser's brain state response may then be presented in reference to themovie. The computer system may identify similarities or differences inthe brain state responses of each of the users, and determine acompatibility match between users based on the determination. Where therespective users are not known to one another, and a high similarity inbrain state responses is determined, the computer system may thenrecommend a friend match between the users. Each user may perform asearch for other users having similar brain state responses to the samecontent, for matchmaking or other purposes. This functionality may beused in connection with an online dating platform for example. As modernsociety tends to be isolated and saturated with media and advertisingthat creates unrealistic expectations about relationships and the natureof true emotions, brain states can help make communication more genuineif one party can see that the other party has a similar emotionaldynamic or is in the state of mind that is compatible to the partymaking selections for chatting. While online dating has had success withmatching algorithms based on self-report, simply the fact that a chatpartner, online date, or even a fellow gamer, has been matched up with auser based on brainwaves will generate the shared context necessary forsmooth social interactions. The authenticity of non-voluntary brainwavesmatching one with a partner or group online will likely motivatecuriosity and openness in an online forum.

In another non-limiting exemplary implementation of the presentinvention, brain state data may be used in conjunction with a wearablecomputing device such as Google Glass™. As shown in FIG. 37, a user maybe wearing a Google Glass-style device including a glasses-mountedcamera. The computing device may be configured to activate the camerawhen a bio-signal sensor detects a particular brain state in the user.The bio-signal sensor may be integrated or attached to the Google Glassdevice, or may be in wireless communication therewith through Bluetoothor other suitable wireless communication protocol. The user's brainwavesmay be monitored for markers of interest. The video camera may bemounted like an eyewear whose camera has the same line of sight as theuser looking straight ahead. When the user sees something of interestthat is above a particular brain state threshold as determined by aprofile or brain state classification rule, the video camera may recordwhat the user is looking at. The device may perform life-logging bykeeping track of interesting things during the day. The device may alsomap the user's location when recording was triggered by something ofinterest using GPS. Brain state information recorded in this may assistretailers, city planners, or others determine what people findinteresting with respect to advertisements out in public spaces,architecture, public performances, political speeches, or other eventssuch as car accidents, etc. The camera may also be activated to recordwhen the user is determined to experience a particular brain stateresponse when seeing, talking to, or meeting another particular person.In this way, first meetings with a future friend, partner, or spouse mayalso be recorded. This implementation may work in the following way. Theuser may first turn on a brain state camera application. The user mayselect to calibrate the application to the user. The rules engine mayselect a pipeline ID to use with the calibration. The user may completea relaxation exercise that invokes a neutral state of mind. The user maycomplete a second exercise that is based on watching an interesting andor entertaining video. A histogram of the distribution of beta brainwave power when relaxed and a second histogram of the distribution ofbeta power when interested are built. A message with the values of bothhistograms can be transmitted to the user profile of the user stored inthe cloud. The level of interest based on beta power exhibited can becalculated based on the probability by using the reference values in thetwo histograms. The brain state camera may now be ready to record. Theuser may select the brain state camera application for active recording.Several pipelines for signal processing, extracting features andclassification may be implemented in the application each identified bya pipeline ID. The application may communicate with the cloud platformof the present invention. The pipeline ID may be selected and may beused to extract features for this application tuned for this person(e.g. the histograms of relaxed beta distribution and interested betadistributions discovered during calibration). The pipeline ID mayinstruct the application to calculate relative beta power from the EEGsignal of the user. The EEG along with time stamps are continually beingstreamed to the cloud. The probability that the user is in an interestedstate based on their relative beta power is calculated based on thehistograms built during calibration. If the probability exceeds athreshold then a signal turns on the video camera. The applicationassigns a session ID based on calendar date and time. The video and thetimestamps of when the recording was started and stopped with eachsession ID and APP ID is transmitted to the Cloud and stored in theUser's profile. The start and stop time stamps are used to identify thesegment of EEG signals associated with the video recording along with anActivity Type called “interest based video recording”. The time stampsare labelled with start of activity, end of activity respectively.Additional information such as the user's GPS location may also betransmitted to the Cloud. New EEG information may be stored in theuser's profile that can be used to enhance the histograms and henceability to predict the user's level of interest. Additional processingcan be done in the Cloud to determine a geographic map of interest.Level of interest and time of day. A plot of interest scores across theday. Analyzing the information in the Cloud across the sessions of theuser can determine behavioural patterns that are helpful to the user andhelp them retrieve videos that he/she has accumulated.

As shown in FIG. 38, in a non-limiting implementation of the presentinvention, the computer system may change an alert notification (e.g.ringtone) of the user's computing device (e.g. mobile phone) when theuser's brain state is determined to be of a certain type. The user'sring tone may therefore reflect the user's mood, or the ring tone may bemodified based on the caller's brain state, where the caller is alsowearing bio-signal sensors in accordance with the present invention. Thering tone may be adjusted to the user's mood so that the call is lessdisruptive when the user is in a particular brain state. A ring tone maybe shut off when it is detected that the user is annoyed or sleeping.However, some filtering or selective choice of expression is neededdepending on the who the called party is as persons express differentlyto loved ones, colleagues, superiors, etc. Ring tones can be used tocreate an audible emoticon.

Modulating a ring tone by emotion over Skype may be provided as follows.An application in accordance with the present invention may classify EEGdata. The application may send classified emotion to Skype through a webbrowser or through the Skype application. The calling party may initiatea call with another Skype user (the “called party”). Skype may send aring tone based on a classified motion of the user wearing the EEGsensors that plays over a speaker of the called party. Furthermore, thepresent invention may provide for a user communicating with another usermay to enhance any such communications with the respective user's brainstate such that both users may be able provided with an indication ofthe user's own or the other user's brain state, or the computing deviceof one of the users may be configured to modify presentation of thecommunication based on one or both of the brain states received. Thearchitecture of the present invention for such an implementation may beseen in FIG. 15.

In another non-limiting example a user may effectively provide a commenton content or shared content without explicitly stating an opinion. Theuser's brain state may be determined and this may be shared to theauthor or poster of the content without the user having to spend anytime composing a comment. In this way, collaboration may occur betweenusers.

Brain State Response to Advertisements

FIG. 39 shows a representation of an aspect of the present inventionwhere a user is presented with an advertisement. The advertisement mayappear on a web page, overlayed on a video, in a game, or on otherdigital content. The advertisement may prompt the user to achieve aparticular brain state in order to trigger an event. For example, theuser may be asked to “get happy” to received a coupon code. The user'sbrain state is determined and transmitted to the advertisement orcomputer controlling the advertisement. If the user is able to achieve ahappy brain state above a particular threshold, the advertisement may bemodified to indicate to the user that the brain state was achieved, andpresent the user with the coupon code.

There is often a need to make a number of decisions in marketing aboutproduct features, style, advertising effectiveness, etc. The status quofor gathering information for marketing is to conduct focus groups,surveys and other means of product testing where potential customers andusers are asked for their thoughts and opinions. Neuromarketing is a newfield of marketing research that studies consumers' sensorimotor,cognitive, and affective response to marketing stimuli. Monitoring auser's brain waves in real-time in response to interactions with orpresentations of advertisements may provide for more information abouthow users react to particular advertisements, or the products, services,or brands associated with them. While a user is viewing anadvertisement, an additional icon or indicator may be shown to the userat the edge of the ad that is an indicator of mood. The user may receivefeedback on the user's mood (e.g. increasing smiley face). Theadvertiser may offer a better coupon based on how strong a brain stateresponse the user is able to achieve. Also, the user may be rewarded toachieve a better mood because when a consumer is in a good mood, theconsumer is more likely to buy the product being advertised, or engagewith an associated brand in a positive manner.

Focus groups wearing bio-signal sensors of the computer system of thepresent invention may also be used for testing audience response to amovie or a television show, or to a product such as a new tabletcomputer or a children's toy. Often directors and producers want totailor movie endings or key scenes depending on the demographics oftheir audience (e.g. European, North American). Directors can testdifferent endings of movies with test-audiences to determine which leadsto the most entertaining ending.

The emotional impact of online advertising may be estimated by analyzinga person's brainwaves. A web browser may present an advertisement.Timestamps associated with advertising events may be output by thebrowser along with the brainwaves of the person. The timestamps allowanalysis of brainwaves occurring at the same moments as the advertisingevents. Events may include: click through; pointer brush; click close;and other events. An example of how this can work is through the use ofinterstitial ads. An interstitial ad displays before a user can accessrequested content, sometimes while the user is waiting for the contentto load. The emotional response of the ad can be assessed for userswearing the bio-sensors of the present invention. It is presumed thatpeople are online while for other reasons and not for the ad itself.Over time advertisers may understand the impact of their content on auser and change the format or type of information or messages that aresent for tailoring to the specific user.

Bidding for Advertising

In another non-limiting example, real-time bidding for onlineadvertising delivery may be based at least partly on brain state, andimplemented by the architecture shown in FIG. 16. Online advertisinguses the internet to deliver promotional marketing information toconsumers. Real-time bidding may be executed using for example one ormore ad networks. Real-time bidding permits selling and buying onlineadvertising based on computer algorithms that analyze metrics of theconsumer. The intent is to increase the effectiveness of advertising byanalyzing data about a specific consumer. In current use today consumerinformation can include, age, gender, geographic location and possiblysearch engine queries. In this invention, the data about the consumer issupplemented with analysis of the consumer's brain state. The brainstate can include longer term metrics over time (e.g. an average ofemotions over a long time period) to shorter term emotions. Emotion,mood and feelings may play an important role in a person's financialdecision making. When a person is in a good mood, the person may be morelikely to be influenced by visual stimulation and hence more likely tomake a decision to purchase. Moreover, a person in a good mood tends tobe more receptive and open to new products, or encouraged to purchasebecause the person is more willing to take risks and the person is morelikely to buy impulsively. Advertisers may be willing to pay a higherpremium for placing ads to users that are in a good mood.

Real-time advertising bidding may occur as follows, with reference toFIG. 47. A Publisher of a web site supports real-time-bidding to placean ad on a per consumer basis. The Publisher is a supplier of ad space.A user clicks on a link in their browser that sends a query to thePublisher's website. The publisher's web server sends HTML code to theuser's browser with the address of where to get the content. The user'sbrowser sends a message to the Publisher's ad server. The Publisher's adserver returns the address (site ID) of a Supply Side Platform (SSP),the publisher's ID and the dimensions of the ad space. The SSP is anexchange that brings together suppliers of ad space (i.e. Publisher) toadvertisers bidding for ad placements. If the user has a cookie IDassociated with that SSP, the user's browser sends a bid request alongwith their cookie ID to the SSP. Large SSPs in the USA have 90% coverageof the US internet population. The SSP starts the auction. The SSP sendsmessages to each advertisers' Demand Side Platform (DSP) with the user'scookie ID, the URL the ad will be delivered to and the user's frequencyof visits is for that site. Each DSP determines the value of placingthat ad by considering the information they have about the user. Thecookie ID sent by the user is searched in the DSP database to find dataabout the user. The DSP sends a bid back to the SSP. The SSP picks thewinning bid and sends the address of the winning adserver back to theuser's web site. The user's browser displays the ad.

Advertisers may be willing to pay more than the normal rate for adimpressions associated with consumers with specific characteristics(e.g. 10 times more than the normal rate). Also, consumers are willingto give up some privacy to brands that they like. The user is aware thatadvertisers will have access to features of their brainwave data. Theymay allow this because of incentives offered by advertisers, or the useris willing because they are offered some control over the content theyreceive. This is particular true of consumers who have become“advocates” of a specific brand.

In exchange for having their brainwave features available toadvertisers, the user may be able to control the quality (i.e. reduceannoying ads), or source of ads (e.g. not receive ads from a particularpolitical party) and control content (e.g. favour ads that have pleasantscenery) of advertising messages that they receive.

In a non-limiting example, real time bidding for online advertising maybe enabled via a sensor application, as shown in FIG. 17. A Publisherweb site has ad space to sell to advertisers. They sell the space basedon a bidding process across advertisers competing for the adspace—real-time-bidding. The Demand Side Platform (DSP) is an exchangethat brings together suppliers of ad space (i.e. Publishers) toadvertisers bidding for ad placements. Data Providers (DP) are connectedin close proximity to a DSP to allow them to return data quickly to aDSP (e.g. 10 ms). Sensors can be biological sensors or other kind ofsensors hosted by the client device (e.g. outdoor air temperaturethermometers, other environmental sensors, accelerometers, light,environmental molecules, ambient sound, wind speed, water temperature,etc.). A user with a client device (computer) with one or more attachedsensors (e.g. EEG headset) opens an application on the device thatmanages connectivity and streaming of data of one or more sensors. Theuser logins into the Sensor Cloud Platform (SCP) with a User ID andpassword for authentication. Raw sensor data is streamed from theapplication to the Sensor Cloud Platform (SCP) with the information asper PCT Patent Application No. PCT/CA2013/000785 (e.g. session id,signal type, channel #, etc.) header in the streamed sensor data. TheAPPLICATION invokes each browser on the client device instructing it tosend an HTTP query to the SCP with its SCP user ID (i.e. John Doe). TheSCP sends a HTTP response back with a set cookie instruction that isuniquely associated with that SCP user ID called SCP cookie (for exampleSCP789 and the SCP user ID in the cookie, e.g. John Doe). The SCPselects a pipeline that classifies sensor data into features/predictionsof interest to advertisers. The SCP applies the Algorithm Pipeline ID inthe User's profile by looking up the User ID to find the correspondingalgorithm pipeline for analyzing sensors associated with online ads. Itextracts features from the user's sensors and applies a classifier topredict a state, e.g. mood. The SCP calculates either the sensorfeatures, its prediction (e.g. good mood) or both. The SCP sends on afrequent periodic basis (e.g. every 100 ms) features/predictions to DataProviders (DP) along with the SCP user ID. Cookies may be synchronizedin the following way. The DSP has previously seeded the internet withDSP cookies using a unique User ID for each browser that has received anad from an auction the DSP has run. Large DSPs have 80-90% reach ratesto the US internet population (See ref.:http://www.adopsinsider.com/ad-serving/diagramming-the-ssp-dsp-and-rtb-redirect-path/).A new entry is entered into the Cookie Matching Table whenever thebidder has won an ad placement. The Cookie Matching Table allows thesensor based ad platform to associate SCP User IDs and hence sensorbased information about that user for future auctions involving theuser's sensors.

The cookie matching table keeps track of associations among cookie IDsbased on previous winning bids. On a previous visit by the user'sbrowser to a web page with real-time bidding for online ads, the DSPsets a cookie in the browser, e.g. DSP456 as shown in the cookiematching table example below. Anytime after the DSP cookie is set, theuser's browser visits a website with ad space for auction, the browseris redirected to a DSP. The bidder enters DSP456 for this auction intothe cookie matching table. The bidder wins this auction and sends an adplacement to the user. The User's browser is informed where to get thead and embedded in this response is the DSP User ID. The DSP User IDuniquely identifies this user across all online ad auctions. The user'sbrowser sends an HTTP request to the Ad server with the DSP User ID. Thead server responds with the ad and the cookie SCP789(John_Doe) and theuser's browser stores the SCPcookie. The ad server associates CookieMatching Table is completely filled based on winning this auction. OtherCookie Matching schemes are available as described in the prior art. Forinstance, Google runs an external Cookie Matching Service (see ref.:https://developers.google.com/ad-exchange/rtb/cookie-guide). Matchedcookies allows the sensor based ad platform to associate future bidrequests with a particular SCP based User ID. This will allow subsequentad auctions that involve this user to use sensor information to make aninformed decision for bidding on an online ad.

An example of a row of a cookie matching table may include: DSP UserID=DSP456; SCP Cookie ID=SCP789; SCP User ID=John_Doe; and SignalType=EEG, Heart Rate, etc.

A method implementing the architecture of FIG. 17 may be as follows. Theuser clicks on a link in their client device browser, e.g. www.acme.org.Client device sends an HTTP query to www.acme.org (denoted in FIG. 17 asreference (1)). At reference (2), the Publisher's website sends theaddress of a DSP with the publisher's ID and the dimensions of the adspace to the client device browser. At (3), the client device sends itsDSPcookie to the DSP. At (4), the DSP sends a request for data to itsdata providers with the DSPcookie. The DP looks up the DSP cookie in theCookie Matching Table and finds the associated SCP User ID of this user.At (5), the DP returns the sensor features of one or more Signal Typesof the SCP User ID, as calculated by the SCP, to the DSP. The featuresmay be real-time or an historic average or aggregation across a longperiod of time. The DP has 10 ms to return its response. the DP hascached data from the SCP indexed by SCP User Id and calculated featuresor predictions of the sensor data. This cache allows the DP to respondquickly to queries since features are based on features/predictions thatare already calculated. At (6), the DSP starts the bidding process bysending calculated features/predictions to bidders along with the DSPUser ID associated with the browser that initiated the web request tothe publisher. At (7), each bidder determines the value of placing thatad by considering the information they have about the user including thecalculated features/predictions (e.g. mood classification). The biddermay also have historic features of the user. Other information of theuser is searched in their database to find all data available about theuser. In the case that the sensor is off, the bidder may want toconsider historical information that it has received and or stored aboutthe user. The bidder sends a bid back to the DSP. At (8), the DSP picksthe winning bid and sends the address of the winning ad server back tothe user's browser. At (9), the user's browser sends an HTTP request tothe ad server. At (10), the ad server fills the ad space and sends thegraphics animation etc to the user's browser to displays the ad.

An advertising system can determine whether or not the user clickedthrough. The SCP is still receiving sensor data from the User. Duringand after the ad has been placed, the SCP calculates sensorfeatures/prediction based on the sensed reaction of the user. Over timeadvertisers will understand the impact of their content on a user andchange the format or type of information or messages that are senttailored to reaction (e.g. emotion) of the user.

In a non-limiting example, real time bidding for online advertising maybe enabled via instantaneous reading of sensors, as shown in FIG. 18. At(1), the user clicks on a link in their client device browser, e.g.www.acme.org. Client device sends an HTTP query to Publisherwww.acme.org. At (2), the Publisher's website sends the address of a DSPwith the publisher's ID and the dimensions of the ad space to the clientdevice browser with JavaScript to determine the sensor id and sensordata. At (3), the JavaScript runs on the client's browser and it getssensor id and sensor data/features. At (4), the client's browser sendsthe sensor id and sensor data/features to the DSP. At (5), the DSPstarts the bidding process by sending sensor features to bidders. At(6), each bidder determines the value of placing that ad by consideringthe information they have about the user including the brain featuresand mood classification. The bidder may also have historic brainwavefeatures of the user. Other information of the user is searched in theirdatabase to find all data available about the user. In the case that thesensor is off, the bidder may want to consider historical informationthat it has received and or stored about the user. The bidder sends abid back to the DSP. At (7), the DSP picks the winning bid and sends theaddress of the winning ad server back to the user's browser. At (8), theuser's browser sends an HTTP request to the ad server. At (9), the adserver fills the ad space and sends the graphics animation etc to theuser's browser to displays the ad.

In a non-limiting example, real time bidding for online advertising maybe enabled via streaming sensor data requested by browser JavaScript, asshown in FIG. 19. This option may require cooperation from a Publisher'swebsite to embed JavaScript to redirect sensor streams to sensor cloudplatform. At (1), the user clicks on a link in their client devicebrowser, e.g. www.acme.org. The Client device sends an HTTP query towww.acme.org. At (2), the Publisher's website sends the address of a DSPwith the publisher's ID and the dimensions of the ad space to the clientdevice browser. The Publisher's web site also sends JavaScript to theuser's browser. At (3), the JavaScript runs in the user's browser. TheJavaScript obtains the sensor id and sends it and streams sensor rawdata to the sensor cloud platform. At (4), based on the HTTP codedelivered in step 2. The user's browser sends DSPcookie to the DSP. At(5), DSP sends a request for data to its data providers with theDSPcookie. The DP looks up the DSP cookie in the Cookie Matching Tableand finds the associated sensor ID of this user. At (6), The DP returnsthe sensor features of the sensor id as calculated by the SCP to theDSP. The features may be real-time or an historic average or aggregationacross a long period of time. The DP has 10 ms to return its response.the DP has cached data from the SCP indexed by sensor id holdingcalculated features of the sensor data. This cache allows the DP torespond quickly to queries since features are based on features that arealready calculated. At (7), the DSP starts the bidding process bysending sensor features to bidders along with the DSPcookie associatedwith the browser that initiated the web request to the publisher. At(8), each bidder determines the value of placing that ad by consideringthe information they have about the user including the brain featuresand mood classification. The bidder may also have historic brainwavefeatures of the user. Other information of the user is searched in theirdatabase to find all data available about the user. In the case that thesensor is off, the bidder may want to consider historical informationthat it has received and or stored about the user. The bidder sends abid back to the DSP. At (9), the DSP picks the winning bid and sends theaddress of the winning ad server back to the user's browser. At (10),the user's browser sends an HTTP request to the ad server. At (11), thead server fills the ad space and sends the graphics animation etc to theuser's browser to displays the ad.

Feedback on Performance

There are many situations where people seek to feedback on theirperformance either during practice, preparation or rehearsal or during avenue where a person is performing a task with upon or in front ofothers. This can include a number of categories such as business peoplemaking a sales presentation or pitch to investors, an educator in frontof a classroom, a business person making a report during a meeting orteleconference call, a healthcare worker providing a service for anotherperson. Brain state can add a deeper dimension to feedback by elicitingfeedback that a person themselves may not be able to voice such as thedepth of focus, moreover minimizing bias that people may have. Forexample, you could note the parts of a presentation that are interestingby monitoring P300s, both their presence or amplitude, or other eventrelated potentials (ERPs). You could assess audience member's emotionalstate, or like/dislike (using frontal alpha asymmetry, for example), orlevel or focus or attention (using, for example, frontal beta, frontalmid-line theta, or other measures), or level of drowsiness/alertness.This could be assessed for a single audience member, or averaged acrossa number of audience members. For example, a teacher performance may bebased at least partly on brain states of the students. A new teacher candevelop the teacher's teaching style in front of students by monitoringthe efficacy of the teaching approach on the attention garnered from thestudents. The effectiveness of an investor pitch may also be testedprior to making the presentation to actual investors, by using brainstate data of an audience receiving the test pitch. An example ofperformance feedback for public speaking is shown in FIG. 41.

There may also be applications for speech writing. A politician orexecutive may wish to test and optimize a speech by examining theemotional impact their speech is having on a test-audience at particularpoints in time. A test-audience wearing headsets including bio-signalsensors of the present invention may listen to a draft version of thespeech. Their emotions are captured at each point in time of the speech.The test-audience emotional response is compared to the intendedemotional profile that the politician/executive wishes their audience toexperience. With an understanding of the differences of the actualcompared to the intended emotional responses, the speech can be modifiedto optimize emotional response.

Public speaking is a major source of anxiety in people, and it is alsoone of the most important transferable skills to be successful in themodern world. whether you are in business, politics, academia, or evenservice, your ability to communicate in front of people defines yoursuccess to some degree. The ability to use brain state to get genuine,involuntary responses sidesteps the usual problem of politeness inself-evaluation (especially compared to conventional clicker responses).The fact that public speaking and communication for leadership can onlybe rated by subjective observers makes it very difficult to evaluateone's performance in any reliable way. Brainwave technology can overcomethis obstacle by its reliance on algorithms which can help determineobjective response. In addition we have a precise temporal mapping ofemotional responses that given manual systems that depend on userspushing a button is more accurate.

An exemplary method of applying brain state feedback for public speakingis described. A user may be engaged to speak at some upcoming event. Theuser works on a first draft and gets to a first version of the speech.The user may decide to solicit feedback from some group (peers, team,hired sample, etc.). Group members arrive and are given bio-signalsensors of the present invention to wear of the computer system of thepresent invention. The user may connect a multi-bluetooth peripheral toa computing device. The user starts data collection on the application.The application may detect connection problems or signal qualityproblems, and the user is informed and instructed on how to fix issues.The application may facilitate calibration of emotions to brainwaves foreach user's individual brainwave profile. The user opens dashboard andsees data coming in. The user hits record and delivers a speech to thegroup. When completed, or at any time, the user may stop recording thedata. Multiple interaction modes may be provided including: User canread speech text with emotags compiled from group embedded into thedocument; User can dive into individual or subset of participants; Usercan listen to audio recording of speech with audio manifestations ofbrainwaves overlaid; User can visualize brain data over time; and Usercan export data, audio, visualization etc. Data types may include:emotion (arousal, valence); focus vs distraction; and focus vsdistraction can filter relevance of emotional data to speech. FIG. 42shows an example of the user playing back a speech and viewingindividual brain state responses of speech attendees. Optionally, thebrain state responses may be used to modulate the playback of thespeech, by varying the volume in accordance with variations in the brainstate responses, or by mixing in additional sound effects representativeof moments in the speech that provoked particular responses from theaudience.

Other applications of receiving brain state feedback on performance mayinclude: Marking School Grades (e.g. A professor wears a headband whilemarking student tests assignments to detect if they have any biastowards students, the order of marking or if fatigue is setting in);teleconference monitor (e.g. The participants of a teleconference callin an organization can be monitored to help the person with the “floor”determine if their audience is fatigued, bored, and or distracted);Masseuse (e.g. A masseuse can monitor the person receiving the massagefor levels of relaxation); Actor's Studio. (e.g. an actor can improvetheir performance if they have feedback on the actual emotions they arefeeling. If a scene requires sadness, then the depth of their sadnesscan be reflected to them by the computer system of the presentinvention); Learn Skills by Gaming; Surgical training game; Driving testgame, and others.

The present invention may also be used to appraise a person'scapabilities to assess a level of competence (e.g. by testing composureunder stress of a candidate in a job interview of a stressful job suchas an air traffic controller or call centre employee; or in a drivingtest to track emotions of the person undergoing the driving test).

Augmented Reality Applications

Elements of a scene as viewed through a camera/Google Glass may bemodified to appear based on brain state/emotion. For example, if thesystem detects confusion, a help button can come up. If the user looksat someone the user loves, the EEG could detect the brain state of love,face tracker could detect the face of the person in front of the user(or even do face recognition off of Facebook), and hearts can swimaround the person's head on the augmented display.

Other Examples of Modulating Content

Various other examples of modulating content, or modifying presentationof digital content will be described.

Modifying Display of Desktop Icons

With reference to FIG. 43, a desktop of icons may be visible on theuser's computing device. The display of the icons may be modulated torearrange the icons on the desktop by the user's brain state. Forexample, if the user is determined to be calm or sedentary, all iconsmay be visible. If the user is determined to be focused or moving,irrelevant icons to that brain state may appear faded. Likewise, if theuser is determined to be stressed and sedentary, other icons irrelevantto that brain state may appear faded. Accordingly, the system of thepresent invention may fade irrelevant icons to the user's currentcontext. The bio-signal sensor(s) and an accelerometer connected to thecomputing device may communicate with the computer system to detect thestate of the user and modify the home screen. Other modulations may alsobe possible based on a combination of brain state and accelerometer. Inparticular it may be beneficial to the user to minimize the visual loadpresented to the user when the user is stressed, or when the user ismoving, or both.

Encouraging Homework Completion

In another non-limiting example, the user may set a goal to finishhomework that has a particular deadline. The user's brainwaves may bemonitored by an EEG headband. The system may provide generic rules thatapply to all users when they first use the system. The classifierdetects the event that the user has become drowsy. The following may bea generic inference rule that applies to all users when they first startusing the system: Event=“Drowsiness Detected” IF Condition=CriticalActivity THEN ACTION=“Play Stimulating Music”. The user may respond byinputting a manual override command (presentation control command) andturns the music off. This manual intervention is kept in a log with thisrule that includes all of the information at the time such as date andtime, algorithm used to detect drowsiness. Because of the User's Manualintervention, the ACTION part of the rule may be changed to poll theuser's brain or emotional state and more details about the activity. Thenext time the rule is applied, the system may perform a brief survey ofthe user's emotional state. The survey determines that the user isapprehensive and they are doing math which requires a great deal ofconcentration. The survey also determines that the User's preferredaction is to get coffee. After getting this self-report, the EEGassociated with this event is labelled as stress and added as trainingdata in the User's Profile to the data mining improver. Another rule isadded to the Rules Engine. This rule is added to the User's Profile as adiscovered rule, and may be defined as follows: Event=“DrowsinessDetected” IF (Condition=Critical Activity) AND (Condition=ConcentrationRequired) THEN ACTION=“Suggest Getting Coffee”.

Fostering Creative Writing

In another non-limiting example, the user may set a goal to create a newwork of fiction writing. The user may be facing a blank page and maywant feedback on what thoughts or feelings are holding the user back inthe creative process. The user wants to use brain training tools toachieve a state of innovative thinking and creative ideation. The methodis to detect when the user is having constricting thoughts and oremotions that are getting in their way through mindfulness practice.Mindfulness is the awareness that arises when one places their attentionfully and non-judgmentally in the moment. Awareness can be of thoughts,feelings and physical sensations. In this example, the user is new tomindfulness practice and hence the generic rule that is applied is onethat tries to improve the user's self-awareness of breath throughmindfulness based practice. The generic rule may be defined as follows:IF (Condition=Started Creative Activity) (Condition=Novice MindfulnessPractitioner) THEN ACTION=“Suggest Starting with Mindfulness on PresentMoment”. The action performed by this rule is to provide feedback to theuser when either agitation or drowsiness are detected. This exercise istraining towards a more creative state of mind. After the user hascompleted this exercise they will be a more creative state of mind butmay still slip back into undesirable states. When the user shifts tocreation, the system sets up a timer for periods of time that will ringa chime when the brainwave detects that the user has sustained a periodof drowsiness, agitation or exceeded a time limit. For that duration oftime the user's brainwaves are monitored and then feedback is given tothe user. The user also provides a self report of the quality of theircreativity after the chime rings. The emotional profile of this timeinterval is provided and the user is asked to drop into another mindfulexercise. This will replenish the user's creative reserves and allowsuser to act in full velocity. In addition, this information is added tothe User's Profile. This data adds to the training data allowing thedata mining improver to analyze the brainwave conditions associated withcreativity, drowsiness and agitation to improve the quality of feedback.This rule is expressed as a nested rule as follows: IF(Condition=Creative Activity) (Condition=Novice MindfulnessPractitioner) THEN ACTIONs=[IF “LENGTH AGITATION >5 minutes” OR “LENGTHDROWSINESS >3 minutes” OR “Elapsed Time >10 minutes” THEN “Play Chimeand ask user if they want to do a separate refreshing exercise OR “keepworking with breath guidance” OR “no interventions please”].

The user may respond to this request and provide feedback that they wantto keep working with breath guidance. This preference is updated and therule is updated to automatically do this action until the pattern ofUser's manual intervention indicates to change the intervention.Accordingly, the rule may be defined as follows: IF (Condition=CreativeActivity) AND (AGITATION OR DROWSINESS DETECTED) THEN ACTION=“keepworking with breath guidance”, IF “AGITATION” Then ACTION=play soothingmusic AND unobtrusive audio instructions to a breath guide to adjustbreathing in real time for long, calm, slow, smooth breaths, IF“DROWSINESS” Then ACTION=play stimulating music AND unobtrusive audioinstructions to a breath guide to adjust breathing to Very Deep Breaths.

In this intervention, as agitation is detected then a rule fires. Thesystem may play music that either soothes or creates a higher state ofalertness, and there could be a breath guide to adjust breathing in realtime. If drowsy then deeper breaths may be encouraged with stimulatingmusic.

In addition, the frequency and quality (or score) of these sessions hasbeen kept track in the User's Profile. If the User reaches a certainlevel of mindfulness practice then they will have a moderate expertstatus as expressed by the following rule: IF (User=Brain state ofMindfulness=Moderately balanced) THEN ACTIONs: “USER=ModerateMindfulness Expert”.

Mood Management

In another non-limiting example, applications in accordance with thepresent invention may be provided that enable users to manage theirmood. For example, a user may select that the user wants to be happier.A brain state application may be launched and a base line recording maybe made of the user's brain state. The application may suggest a numberof ways that the user could get happier, including using a number ofapplications that modulate content based on brain state and the rulesengine of the present invention. The user may select to get happier bywatching movies. In the video application, the user may click on a “gethappy” link so that the rules engine is notified of the user'sintention. The video application may recommend putting on headphones towatch the intro to happiness video (e.g. be shocked with a sound andlight show). The short intro allows a baseline reading to be made and italso prepares the user to get happy watching videos. In other words, therules engine in an evaluation mode may create stimulus as part of theget happy video help determine the user's brain state.

With the data collected from the calibration video and the user'sprofile, a number of movies to watch may be recommended that should makethe user happy. The movies are summarized with a sequence of thumbnailsspanning the timeline. The summaries are tuned to the user's profilebased on their and other similar user's reaction to the movies. Therules engine may modulate the video content in accordance with happymode, and present results to the user.

As the user watches the movie, the movie content is modulated in realtime in order to tune it to the user's brain-state and to help providebetter estimates of brain-state when will be helpful for producingbetter modulations and content recommendations. The rules enginemonitors the user's brain state and controls the modulator.

Inputs received from the user where the rules engine receives the userinput requests and implements the appropriate state change include:volume up and down, FFwd, Rwd, Skip (e.g. inputs that builds on theprofile for the user and the particular media); Modulation up/down (e.g.inputs that change the intensity of modulation to meet the user'spreference, and is an input that control a one or multi-dimensionalproperty); and Modulation tuner (e.g. inputs that change modulationproperties in a multi to multi dimensional mapping).

When the movie finishes, or the user presses skip, results may bepresented to the user and the rules engine may control results if theyserve the end goal. Self report questions may be asked that facilitatethe mood transition and the rules engine may control inquiry. New moviesmay be recommended that use the updated information of the person's moodand real-time reaction to the content that just played and the rulesengine may suggest the content.

Where the user is interested in managing the user's mood, the user maywish to stabilize the user's mood, preserve a particular feeling, orstay in a particular brain state once achieving the brain state. Othergoals may include: keeping yourself from dropping into undesirablefeelings; a safe place to experience an particular negative state and tooffer a process for finding your way out (detachment); mood altering;intensifying mood (make movies more exciting); and changing from onemood to another.

The rules engine may need to be able to build up (cluster) states thatthe user experiences, without necessarily labelling the states. Therules engine may then also need to provide a means of labellingunlabelled states/clusters of states.

User State as it Relates to Content Creation

When the system of the present invention is used to enhance the contentcreation, or content editing, process, consideration of the differentphases of the content creation process may be considered. For example,the user may not want any content modulation during a phase where theuser is formatting a document, adding headings, etc. The user may wantto record the user's emo-state during the creative process, but notduring the composition or editing part of the process, since thepermutation of language or word hunting may bear a likeness to theemotional component of the content. Accordingly, in an aspect of thepresent invention, the system of the present invention may determine aphase of the content creation process in which the user is engaged, andconsider the phase determination when applying any modulation rules. Forexample, the brain-state of the user may be determined by the system ofthe present invention prior to the user beginning to actively create thecontent, while the user is only thinking of what to do.

A user may go through various activities in the content creationprocess. For creation of a text document, the user may imagine, wherethe user is creatively ideating on plausible content. The user may addcontent through composition (e.g. converting mental idea into writtenform), editing (e.g. editing the content to improve its effect), oremotifying (e.g. adding brain-state content as a discrete process,either through the addition of symbols (like emoticons), or modulationof existing content in the document (painting text with feeling-addingcolor, modifying font, animating the text, etc.)). The user may feel, byperceiving composed content to consider its effect. The user may beinspired by considering other content to provide material for the user'simagination/ideation. Inspiration may also involve content modulation asit may be used to help the user enter or stay in target states,including: Listening to music; Watching videos; Reading content; andTaking part in online conversations/collaborations. The user may alsoexperience inspiration and not want content modulation, when organizingcontent (e.g. Non-linear busy work, containing actions/additions like:content reorganization, titles/headings, references, links). The usermay transition between the imagination, inspiration, feeling, contentaddition, and organizational phases of content creation.

The system of the present invention may attempt to estimate the phase ofcontent creation, based at least partly on the user's keyboard or UIusage patterns. Optionally, a statistical model of pattern analysis maybe improved through monitoring the user's manual selection of phases,and self-annotation. The system may monitor what words are beingentered; where the cursor is moving throughout the document; and whatpart of the document is in view. Estimated brain or bio-state may alsobe considered when determining the content creation phase, including:brain state; facial expression monitoring; muscle tension; eyetracking(e.g. one or more methods such as EOG, optical, visual evokedpotentials); patterns of usage of the keyboard (e.g. keystroke rate);breathing rate; and body movements as estimated by accelerometerreadings or video analysis.

User Interface for Content Creation

A user interface of an app implementing aspects of the present inventionmay provide controls that allow the user to effectively capture theuser's emotional state or brain state, or “emostate” relative to thedifferent phases of content creations, and to apply the emostate to thecontent either in real-time, or retroactively.

Some system and UI functions may include monitoring and storing user'sbrain state. The user also may be provided with control to allow theuser to retroactively record the user's brain state from some time inthe past (e.g. the last minute). This may be important because the usermay realize that how the user was feeling in the last moments wererelevant to the content that the user is creating. It is hard to feel oncommand, and easier to recognize how you have been feeling, and how youare feeling in the moment.

The UI may provide control to allow the user to select a phase ofcontent creation so that the system know how to interpret actions andapply modulation rules. Some phases that the user may want easy accessto in the interface include: feel; compose; edit; organize/nomodulation.

In the feel phase, the system may record the user's brain-state so thatit can be post-applied as a modulation. For example, there may be anediting step where the user selects the relevant portion and/or modifiesthe content of the recording before applying it to the content.Brain-state estimates may be made using the prior knowledge that theuser is simply reading or considering what has been written. Forexample, in the case that the user has the user's eyes closed, or ifthere is little saccadic movement the system may assume that the user isconsidering rather than reading.

In the compose phase, the system may have prior knowledge of how tointerpret the user's bio-signals based on previous statistics and orcalibration that the user has performed to help train the system. Withrespect to calibration, the user may read a short passage, and thenrewrite from memory, or the user may be shown a picture that is to bedescribed in writing. The set of calibration exercises may also may bedesigned to elicit emotional responses so that they can be moreeffectively detected during a writing composition task. The system mayrecord the data for doing post-modulation or if the user has selectedreal-time modulation. The system may modulate the text that is beingentered using the user's estimated emo-state. Optionally, the system mayanalyze the user's bio-signals to estimate the user's emo-state, usingaffect priors that come from analyzing the word that the user hasentered for their emotional content.

The system may include a word analyzer component implemented in hardwareor software, that extracts context and semantics about text. The outputof the word analyzer may supply prior probabilities to the classifierabout probability that the text conveys an emotion or some other state.This prior probability is sent to the classifier. The classifierincludes the prior probability (e.g. using Bayes Rules) to estimate theprobability that the user has a particular emo-state.

In the edit phase, the functions may be similar to the compose phase,except that emo-state priors may also come from the content in theregion that the user has selected for editing). If real-time modulationis selected, modulation may be applied to the neighbourhood of thecursor using a curve fitting method, to reveal to the user what theuser's current state is, and how added content could affect the documentthe user is adding to.

In the organize/no-modulation phase, emo-state is not considered, sincethe user in engaged in a task that is not-conducive to state estimation,or the user feels that the user's state is not relevant in their currenttask.

Emo-State Editor

Brain-state “emo-state” may be embellished using an emo-state editor,which may be provided in accordance with a non-limiting aspect of thesystem of the present invention, that allows the user to view the user'sstate at some point in the past, on a graph where time is one axis ofthe graph. As the user scrolls and zooms through the timeline, a contentdisplay window can also display a review of the user's actions at thatpoint in time. For example during that reading of a document (where theuser is in the feeling phase), the content display window may displaywhat part of the document the user was considering at that point intime. There may also be a preview option that allows the user to see themodulation effect using the region the user's has selected. Theemo-state editor is also used for content tagging, as well as contentcreation. For example, if another user has shared some content (e.g.text, video, audio, other) and the user would like to tag or commentusing the user's emo-state, the user may record the emo-state and selecta portion that may also embellish or be annotated to the content as aresponse. For example, a user may share a picture to a group of users.Members of the group may respond to the video using short emotags (e.g.perhaps 6 seconds of dynamic emostate). Emotags may be summarized and beincluded in the display of the content, similar to likes and commenttags currently are in many content sharing applications.

The emo-state editor may allow the user to edit or embellish the user'semo-state. In editing, it may be provided to select a region that isrelevant to the content that the user wants to modulate. Inembellishment, the emo-state estimator may be producing estimates of theuser's level of focus, and present it on the graph, The user may recallthe user's affective state at the time, based on the memory cue of thepresented time-line and content display. The user could then embellishthe graph with the user's affective state by dragging an emoticon ontothe timeline

An application of the emo-state edited using the editor may include, ina content experience with a definite timeline, such as a video or audioclip, or with text where text-to-speech or a scrolling display windowmay be used, the association between content and emo-state can be madedirectly without user intervention. In an asynchronous experience theuser may need to apply the emo-state timeline to the content manualusing a time-warping operation. A region of text may be selected toapply the modulation. In the option where the text is consideredsequentially as a timeline in itself, the emo-state may be appliedlinearly, or with time based control points that allow for a non-linearwarp to align the writing content with the emo-state content modulation.Sections of text may also be considered as a block and the emo-state canbe used to modulate it using a rules such as: apply the averageemo-state; apply the most extreme emo-state; apply the most-commonemo-state; and apply an emo-state timeline as a dynamic modulationthrough animation (e.g. making the letters dance to your emo-state).

The UI may also provide a means of editing the emostate or blending itwith other states in the case that portions of the document arereorganized or the user inserts content.

In an example, emo-state content modulation may be blended with existingmodulated content using curve fitting. Typically the curves are fit suchthat the modulation transitions are characteristic of the user's typicalbrain state dynamics. For example, instantaneous transitions are notseen in emotional state. Emotional transitions take time, where speeddepends on the states the user is transitioning between, the stimulusand the individual. The user may also be presented with manual controlto affect the blending operation (e.g. the transition steepness), asthere are many instances where the user's own real-time state dynamicsare not ideal, which may occur, for example, in the case where thecontent is related to a discontinuous experience (e.g. like a scenechange in a story), or where the content may relate to an experiencewith discontinuous or non-uniform, or accelerated time.

Youtube Enhancement

In an example modulation of video content, the portion of a video thathad the highest positive emotional impact as classified by the user'sbrainstate may become the thumbnail to represent that video.

An enhancement of video content may include video playback speedenhancement. The user may choose to view a video clip with brain-statespeed enhancement, where portions of the video that are un-interestingare played back at a higher speed so that less time is wasted (e.g. allof the clips will be visible but played faster). The user can go back atany time and watch the clip at regular speed. Playback speed isestimated using user's current brainstate combined with the priorinformation from brain-state observation statistics from other likeusers, or the current user if the current user has seen the contentbefore. User likeness may be determined through user profile analysis asdescribed herein. The user may provide manual inputs into the playercontrol for controlling playback speed, and these manual inputs mayimprove any playback speed estimation rules as well as allow the user tohave manual control over speed when desired.

Modulating Presentation of Content—Examples

Many ways to modulate presentation of content in accordance with aspectsof the present invention may be possible. Further examples or detailsregarding examples previously discussed are now provided.

Training

Where the user goal is to get better at something, several activitiesmay be available. For example, the computer system may provideprescribed or guided exercises to improve the user's ability at theactivity. The user may select a specific goal that may be suggestedbased on data that has been recorded. Exercises may also be suggestedbased on the goal. Different exercises may be appropriate given thecontext or media. The user may want to train with videos, audio, videogames, or interaction with other users.

Neurofeedback exercises may be provided where input is taken from theuser's bio-signal sensors and other inputs to create or modulatestimulus in real-time that allows the user to learn. The learning may bereward-based learning based on specific brain states. Rewards could bebased on different channel than the stimulus (e.g. watch movie, and getpopcorn when you are in the right state), or the reward could be in thesame channel as the content (e.g. watch movie, and it becomes clearerwhen you are in the right state).

Training procedures may not require real-time feedback but are comprisedof a sequence of steps, and involve content.

The rules engine (modulation controller) may operate to use the outputof a signal processor to generate the right real-time feedback toachieve the user's goal, and gather information about a user doing anexercise.

Creation

Where the user goal is to create content, the creation activity mayinvolve feeling, imagining, composing, editing, and getting inspired.Goals of the rules engine may include: providing a short exercise to getthe user to start the content creation activity; modulate content sothat the user is encouraged to continue the content creation activity;embed brain state information into the content; generate creativeinsight; getting feedback on contact from others; learning to optimizethe different phases (activities) in the creative process; creatingcontent that depends on brain state; discovering what is blocking ordampening creativity.

The rules engine (modulation controller) may need to be able to estimatewhat activity of the creative process the user is engaged in, so that itcan do the right thing (like give real-time feedback or not). The rulesengine may need to be able to take user manual input of what Activity ofthe process they are in (e.g. (Feeling, Imagining, composing, editing,getting inspired). The rules engine may need to take user input abouthow the user would like the system to function in order to achieve theuser's goals (e.g. start/stop/edit an emo-state recording). The rulesengine may use the output of signal processor to generate the rightreal-time feedback to achieve the user's goal.

Social

Social goals of the user may include: enhanced communication (e.g.easier; deeper; and faster, etc.); entertainment; inspiring; regulatingmood (e.g. detach yourself from intense mood that may come from a socialinteraction which is usually negative but could also be related topositive emotions as well); methods for getting in the right frame ofmind for relating with others (e.g. exercise interventions, real-time);methods to put two or more people in greater synchrony (e.g. exerciseinterventions; real-time); generating an extra band of communicationthat could help overcome miscommunication; validate communications;non-verbal communication; and non-verbal ways of being present withanother or a group.

Corresponding goals for the rules engine may include: match other usersbased on the emotion brain state of a single user; match other usersbased on type of shared goal; compare brain state of one user to anotheruser (this can be in real-time or asynchronously); create real-timemodulation to represent the state of one or more people simultaneously;and create real-time modulation to help people get more in sync witheach other.

The rules engine may provide for collaborative brain art/music or acollaborative brain game to achieve the same brain state or to do abrain obstacle course together. The rules engine may provide formultiple users to listen to music together and render real-timeengagement or affective state. The rules engine may uses the output of asignal processor to generate the right real-time feedback to achieve theuser's goal. The rules engine may use a semantics engine to examinewritten or verbal text to get an emotional tone, and compare thisemotional tone to estimated emotion of user to determine genuineness ofcommunication.

The rules engine may need to know what the user is doing, is respondingto, or considering (e.g. which messages in a chat), such as whether auser is doing something unrelated to an emo-conversation.

The rules engine may connect with a social media web site and managecommunication therewith.

Content may be recommended driven by emotion. The user's response torecommendations or viewed recommendations, may cause the rules engine toadjust recommendations for the particular user, as the current emotionof the user may be determined to have changed based on the contentreceived.

Content that a user posts may be tagged with the user's emotion whenthey created they content (Emotagging). The user may have access toemotagged events/posts of the user's friends. A user can search or sortby event and or state.

Personal content may also be tagged by emotion or brain state. A user'ssocial media history may be crawled while recording a user's response.Content is curated that has emotional meaning to user.

The user may also find chat partners that feel the same as the user.

Health Management

In a health management application, goals may include: monitoring acrossa number of variables; diagnosis (physiological state estimation);asynchronous feedback based on statistics (suggestion to change abehaviour); recommend neurofeedback based therapy; and administerneurofeedback based therapy.

The rules engine may have an evidence database with probabilities ofconditions based on measured physiological symptoms plus demographic anduser self-report. The database may be tied to personal health recordswhich is an external repository of health information.

The rules engine may estimate a diagnosis based on aggregatingprobabilities found by measurements and other variables such as applyingBayes Rule.

The rules engine may have a list of recommended strategies andprioritized by cost of level of effort and find common advice that isthe root of many problems (e.g. get more exercise, eat more healthily,get plenty of rest, manage your stress better).

The rules engine may track and monitor progress of the user over time todetermine patterns of improvement, or rates of improvement or decline(e.g. longitudinal analysis).

An application may include managing/treating a psychological disorder.In this case, a user's brain state may be tracked. Treatment may includethe user engaging with exercises that can improve the user's brainstate. Performance in the treatment is monitored by the rules engine andmay recommend exercises that engages the user. Neurofeedback may beprovided for depression or anxiety.

Another application may include early warning of panic attack. A warningcloud may pop up when the user is showing signs of receding into a panicattack. Content can be switched into something that is soothing orrelaxing.

Another application may include emotag journal content. A user'semotions may be detected and recorded. These emotions may be used to tagemotionally strong passages that the user may not even be aware of. Textto speech conversion reads the journal back to the user modulated withthe emotion that the user felt. The user may experience increasedself-awareness. Content could be shared with a spouse or doctor.

Another application may include adding a brain state to a yoga session.Tension in the mind is related to tension in the body. A user can moreeasily release tension if the user is in the right frame of mind.Guidance can be given to improve self-visualization exercises.

Another application may include a healthcare worker viewing a brainstate of a user. For example, a masseuse may see feedback ofrelaxation/stress of a person receiving massage. The person needs to berelaxed to receive maximal benefits of massage.

Mass Audience Input

For brain state determined for a mass audience, content of a televisionbroadcast may be changed, or a most excited town may receive a filmpremiere.

Employee Management

For employee management, modulation of presentation of content mayinclude: Test candidate composure under stress—objective evidence oftheir brain state; Change employee's environment based on focus orstress; Employee monitors emotional state to prepare themselves formeeting; Employee self-monitors for stress, focus etc.; Employeecommunication enhanced by adding emotional information while theycommunicate to each other; Managers monitor employee mood and systemrecommends productivity/fun actions; and Employee wellness increases byhaving them brain fitness exercises.

Customer Service

For customer service, modulation of presentation of content may include:tech support can see how frustrated user is; priority of callers to callcentre changes by frustration level of caller; brain-sensing on callcentre employees.

Video Games

For video games, modulation of presentation of content may include:emotions change appearance of an avatar—focus/calm could also improveaim, accuracy, strength, etc.-adapting game mechanics; emotions directlydrive game (e.g. provide a game where you have to induce brain states insomebody else like a stand up comedy game; handicap other players' powerif you are focused; if you are sleepy, game difficulty rises); OnlinePoker (e.g. user cannot play without using the computer system of thepresent invention; other player sees emotional states as tells; playermust suppress these as part of the game).

An avatar may represents a user in a virtual world (e.g. 2nd Life). Theavatar can be animated by emotions that are driven by the actualemotions of the user as detected by computer system of the presentinvention. A profile can be set up that maps the actual emotions of theuser to the emotions output by the Avatar. One mapping profile could bethat the avatar represents the actual emotions that the userexperiences. Another mapping could be to mask and redirect emotionsexpressed by the avatar to influence other users in the virtual world.For example, in a virtual game of poker, the avatar would expressemotional responses based on the strength of the hand to facilitatestrategies such as bluffing with a weak hand, or acting sheepish whenholding a strong hand and vary the responses so as not to be predictableby other players.

Education

In an education environment, students may be monitored to determine howthe students feel when being taught material. Monitoring the studentsmay help to determine when to intervene and what to focus on. For eachstudent determine a time of day when the student is most receptive tolearning. This may help the teacher determine when to teach difficultsubjects (e.g. teach math in the morning or afternoon). Monitor how wellstudents are doing (online e-learning). Course material may be modifiedin response to be more difficult or easy.

Monitoring special needs students may provide for early detection of anepisode.

An instructor may determine which course material is effective byexamining brain states of the whole class, and adjust content andlearning style accordingly.

Level of student focus/stress may change course content.

For language training, spoken sentences (cadence, speed, emphasis) maybe re-modulated to optimize recognition of words.

For speed reading, pace and format may be modulated to ensure it iseffective.

Emotional Intelligence

It can be therapeutic for a user to understand the emotion(s) the useris feeling. This could occur in a therapist/patient setting or within asupport group. Emotional intelligence could be taught by compiling thestatistics of emotions across a classroom in response to scenes orphotographs. Another example of classroom teaching is the impact ofmusic on emotion. For example, Daniel Levitin along with KitchenerWaterloo orchestra played a piece of Beethoven to an audience and thenpolled the audience as to which of 7 emotions were evoked. The histogramand statistics of responses were displayed back to the audience. It wasinstructive to learn that 80% of the audience had chosen the sameemotion.

Enhanced Communication Tool for People with Cognitive Deficits

People that have difficulty expressing themselves (e.g. autistics,people with dementia) can use the computer system of the presentinvention to represent their emotions to people around them. This wouldmake it easier to understand a person with a cognitive deficit. Acaregiver can more easily recognize when the person being cared for isupset or agitated but may not have external signs of agitation.

Actor's Studio

An actor can improve his or her performance if the actor can receivefeedback on the actual emotions the actor is feeling. If a scenerequires sadness, then the depth of the actor's sadness can be reflectedto the actor by the computer system of the present invention.

Mood Ring/Mood Headband

A person's emotional state can be displayed as jewelry such as aheadband that has colour and intensity of light to represent theiremotions. The purpose of to have fun in a social setting. This issimilar to the popular mood rings that change colour depending on thewarmth of the user's fingers.

Journal or Diary Entries

Entries in a diary may be enhanced with actual emotional state ascharacterized by the Muse. This could be a good complement to therapiessuch as Mindfulness Based Cognitive Therapy.

Video Modulation

The present invention may be used to provide real-time modulation ofvideo from a camera/augmented reality.

Content Modulation Generally

Types of modulation of content may vary based on the type of digitalcontent. For videos and movies, content modulations may include: trailson motion; reduced saturation (less emotional content, becomes moreaudio centric, perhaps more motion centric; increased contrast, edgedetection, selective blurring.

For audio content, content modulations may include: increased volume;audio event intensification (e.g. increased volume for events; decreasedbackground audio for events to make movies scarier, if it is not intenseenough, or if the user's attention is waning); pitch shifts and discordcreation in order to change the mode (e.g. detune slightly to makemelody droop flat in order to make it sadder); add reverb to increasespatial content and increase immersion; make audio monaural if the useris in a distracting situation or are trying to concentrate on somethingand a high coherence is noticed between the music and the user brainwaves.

For text content, content modulations may include: font face; colour;font animation (motion attracts attention). If the computer system ofthe present invention is used in conjunction with an eye tracker, it maybe possible to determine per-word brain state responses for the user.Words can blink and generate ERPs. The text reader may be presented witha sound accompaniment based on text content and brain state. The readingenvironment could also be modulated in order to achieve a particulargoal.

The present invention may provide positive feedback magnifying the rangeof experience (trip out). Emotional states may modulate the visual fieldwith filters that match the user's mood, thereby accentuating the user'smood. For example, slow laziness can be indicated by rendered motiontrials, silliness can be indicated by fun colors, attention and focuscan be indicated by accentuated edges.

The present invention may also be used to reflect negative feedback tostabilize the experience. For example distraction may be reflected witha narrowing of visual field, and colour may be varied to reflectexcitement or lack thereof.

The present invention may be used to for modulation of audio contentsuch as a recording prepared using a microphone or a musical track. Forexample, enhancements may be provided based on the present invention inorder to magnify the experience by modulating the frequency response bytuning out for example low frequencies and boosting the highs withgreater attention). Musical content for example may be enhanced byintroducing high levels of alpha waves.

The present invention may be used to modulate content that helps a userfall asleep. The present invention may be used to introduce feedback ofthe sound of your own breath and modulate the sound with yourbrain-state. Negative feedback may be used to dampen out emotions. Whena user experiences anxiety, sounds may be modulated to be morereverberant with the reverb model being tuned by a user's specificstate. This draws the mind's attention outwards toward the environmentand away from internal thought. As the mind becomes quiet (emotionalstates conducive to sleep), the sound may be modulated to becomes closerand drier (less processed), producing a sound that is less engaging andmore relaxing and provides a better foundation for sleep entry.

The present invention may also be used to provide a steerable microphonearray that modulates the sound to improve SNR in noisy situations. Anarray can be pointed (either using hard or soft selection) to amplifysounds from a particular direction of interest. A lock in algorithmtracks the user's response using brain-state estimates of auditorystrain.

The present invention may be implemented as a real-time modulationfilter for picture or video capture. The present invention may be usedto embed the emotional state of the photographer, using filters that seta mood, for example for use in conjunction with a photo sharingapplication such as Instagram™.

The present invention may be adapted so that one or more possibleenhancements are presented, and these may be selected or rejected by theuser. Selections may be used to capture user preferences over time.

The present invention may be used to modulate music based on how a firstuser experiences the music, for the purpose of sharing the experiencewith one or more other users. This may be accomplished by modulatingfrequency response (for example by tuning out the low frequencies andboosting the high frequencies, so as to modulate relative attention paidto music segments). Different filter associations may be chosen muchlike presets on a graphic equalizer so that the user's affective stateembellishes the music in a fashion that makes sense given the genre orspecific musical composition.

The context of an activity may also be determined and considered by therules engine when determining a modulation to apply. A microphone may beconnected or in communication with the user's computing device. Thecomputer system may then monitor ambient acoustic noise input capturedby the microphone to determine engagement of ambient environment andcontent. A level of engagement with the outside world may then bedetermined so internal audio may be modulated to mask the ambient noise.

Other non-limiting content modulation applications may include: 1.Compare your brain state to the brain state the Author had when writingthe book; 2. Compare your brain state to the brain state the Musician orSinger had when recording the song; 3. Have your brain state response tomusic also connected to the lights in your home OR project some wickedimages on the laptop or TV screen to enhance the experience in a visualform; 4. For the drawing application, have colours change while theperson is sketching based on their brain state and/or the line stylescould also change from precise and fine to jagged and thick; 5. Couplestherapy: use during sessions to ensure both parties are engaged whilelistening to each other; 6. Picture therapy: use headband and look at aseries of picture to understand which photos evoke desired responses,that way you can choose a screen saver or desk photos that continues topromote those positive states of mind; 7. Eating tracker: to detect howwell you're focusing on your food (which helps with diet) and whichfoods have the most powerful positive responses and also when your brainindicates you're no longer enjoying the food and you're stuffing yourface because that's your pattern (personal experience here); 8. Which TVshows give you the most pleasure or cause the most angst so you don'tget riled before you go to bed so you can sleep better; 9. Long distancedriving exhaustion indicator to use when driving long distances to alertyou when you're unable to focus or are too tired and you need a fewminutes before continuing; 10. Environmental setting tests forAutistics—detecting which situations cause them to ramp upfaster—restaurant with lots of lights and music, dark places, etc. andbeing prepared, eg. having quieting headphones available or othercalming activities; 11. Emotional testing for interviewing emergencyworkers, flight controllers—who can stay calm in test simulations (mightbe good for screening of people who work with kids for particularemotional responses); and 12. Testing optimal work environments (desks,lighting, window exposure, etc.).

Pass-Codes and Security

An individual's brain state or emotion, or other brain characteristiccould be used to unlock or gain access to secure areas, new services, ornew information.

In the security context, this could be providing access to banking,computer passwords or home. In the gaming context, this could beunlocking levels. In the customer service context, this could beimproving your experience or triggering rewards for being nice to thecustomer service person.

Here we also do it with emotion, optionally interfacing with the cloudsystem of the present invention. The user must be in the right emotionalstate in order to do it.

Or perhaps what you are opening is your partner's “heart” (a graphicalrepresentation shown when communicating between users), and when you areboth in the same and “right” state (ie one of compassion, love, sharing,etc), then the heart unlocks.

Digital Secretary

Typically, executives, for example, have secretaries. Individuals whoare hired to support their needs. These are business needs (schedulingappointments, answering phones, sending correspondence), but theyrequire knowledge about the individual and his or her state, needs,desires, emotions, fears, etc. For example, a secretary will know not toput through a call from a certain person if the executive is in a badmood. Or she may know that the executive gets sleepy after lunch, andthat is not a preferred time to schedule meetings. As digital toolsincreasingly replace human actions in our lives, it is important forthese digital tools to continue to know information about the user theyare serving, such as mood, level of alertness, receptivity to newinformation/cognitive load, etc.

EEG and other bio-sensing tools give our digital devices the informationrequired to understand the state of the user, and then make decisions inthe background based on the state of that user.

For example, if the users brain state is beginning to indicatesleepiness by increased slow wave activity, or the presence of markersof stage 1 or stage 2 sleep (K complexes, sleep spindles), the usersdevices could modulate their activity accordingly, such as turning offnotifications, dimming screens, dimming lighting in the house, notputting through calls, etc.

We can use the cloud to track state long term, build heuristics andmodels for the person of what they want, what they want to do at thosetimes, etc., and respond accordingly).

E-Mail or Chat Integration

The user's mail may be organized based on the user's emotional statewhen creating or viewing the email message, which may be associated withthe email message. The computer system of the present invention may tagthe user's email based on how it makes the user feel. An icon may beprovided for each e-mail indicating how much the e-mail made the userfocus. Filters may be provided to modulate e-mail text based on how theuser feels to disempower any negative affect. Such modulation mayinclude introducing words, scrambling words, introducing colours,images, art, sparkles, etc.

When composing an e-mail message, the text may be modulated based on theuser's brain wave data and any corresponding rules.

An emo component to group composition may be provided in a documentmanagement service like Google Docs to allow people to encode how muchthey are focusing/reacting to different areas of the document, like areal-time highlighter. Separate filters may be available that wouldallow the user to view things like how much a viewer was thinking,trying to focus, failing to focus, like, or dislike the document.

With collaborative document creation, it may be good to see theemotional state (brain state as content) of any document collaborators.An activity cursor could also be included to indicate the respectiveuser's affective state (e.g. focused/distracted/frustrated/thinking/etc.. . . ).

Youtube Integration

The computer system of the present invention may interface with aservice like Youtube to provide for recommended video views based on theuser's brain state. The video recommendations may shuffle with theuser's changing brain state.

Media can slow down or pause if the user becomes distracted during apart that is important to the user. For example, a certain character istalking and the user has shown prior interest to this character. Thistype of analysis is made easier if the computer system has access toother people's responses to watching the clip, or if the author oranother software has done a content analysis.

The media can also speed up if the user is bored.

Video thumbnails or preview clips can be auto-thumbnailed (rather thansimply taking the middle of the clip, or auto summarized by yourreal-time emo-state or your brainwave profile).

User input of fast forward or particular video selection can be used forlearning.

When users watch videos together, or at different times brain state datafrom those users may be used to modulate the viewing of the video (e.g.adding in laughter, or boos, or other embellishment that might berelated to watching as a group). The audience watching the video mayalso be rendered around the video frame in real-time.

A crowd sourced editor may be provided that allows feedback from earlypublishing to be visible in the editor to allow you to make bettervideos.

The rules engine could be configured to make a video more or lesspowerful based on the rules engine setting and how the user is reactingto the content. For example, if the user is relating to the sound track,that could be accentuated in parts of the video that others or theauthor found/intended to be dramatic. The soundtrack could be adjustedto maximize entrainment with the user's brainwaves.

Couples' Tools

Resonance between a user's brainwaves and the particular vibrationsproduced by a stimulator could be rendered as a modulated like, ormodulated sound (e.g. sound of breath with modulated overtones ofvibrational stimulator when high coherence is found).

A measure of coherence between both people and the respectivestimulators may be recorded so that both users can see when they are inphase or not.

Entrainment may be rendered in the feedback of the stimulus itself.

Different Types of Sensors

The present invention may be used with various different types ofsensors. Sensors for collecting bio-signal data include, for example,electroencephalogram sensors, galvanometer sensors, orelectrocardiograph sensors. For example, a wearable sensor forcollecting biological data, such as a commercially available consumergrade EEG headset with one or more electrodes for collecting brainwavesfrom the user. For example, bio-signal data may include heart rate orblood pressure, while non-bio-signal data may include time, GPSlocation, barometric pressure, acceleration forces, ambient light,sound, and other data. Bio-signal and non-bio-signal data can becaptured by the internal sensors, external sensors, or both.

The distinction between internal and external sensors refers to whetherthe sensor is contained within a client device such as a mobilecomputing device or not. For example, a smart phone may have a pluralityof built-in internal sensors for collecting non-bio-signal data.External sensors, such as wireless headsets or other types of sensors,may complement or replace the available internal sensors.

Various implementations of the computer system are possible. Forexample, various different applications may be developed that utilizethe computer system and enable different types of biofeedback systems. Afew examples of such applications are provided below, along withexplanations on how these would utilize some of the contributions of thepresent invention.

The present invention may be used in conjunction with various types ofEEG hardware. EEG hardware may be used in conjunction with the presentinvention that has various features including for example:

-   -   (a) a comfortable form that allows the electrodes to make good        contact without pressure points;    -   (b) dry sensor technology, being easy and quick to put on,        comfortable to wear, as well as being stylish so that it can be        worn in public;    -   (c) an on-board 3-axis accelerometer that enables motion input        for games and quantifying head movements; and    -   (d) a wireless connection to a mobile device, for example using        a Bluetooth™ connection.

A headband may be used that produces bipolar readings using AFz as thereference from AF3, AF4, TP9, TP10. A 4 electrode montage may enableestimation of hemispheric asymmetries (with an emphasis on thepre-frontal lobe) and thus facilitates brain-state discrimination thatdepend on this, such as emotional valence (positive vs. negativeemotions).

EEG signals may be over-sampled near 20 KHz, band pass filtered andre-sampled yielding to achieve a selectable output sampling rate from100 Hz to 600 Hz, with 2 uV RMS noise. Active noise suppression may beachieved with a DRL-REF feedback configuration using centrallypositioned frontal electrodes.

Input range of an AC coupled signal (low cut-off at 0.5 Hz) may be 2 mVp-p.

The EEG hardware may use four electrodes, giving better signal to noiseratio than single channel systems for aggregate measurements and thusbetter access to high-beta and gamma bands (fast oscillation EEGsignals), which are particularly important for analyzing cognitive taskssuch as memory, learning, and perception.

The present invention may include a client computer program component.The client computer program may receive the EEG data and input form oneor more sensors (such as an accelerometer or heart rate monitor) as wellas application state and time markers from application-generatedstimulus in the form of audio, video and, vibration. The collected datamay be time synchronized to support ERP analysis, stimulus entrainmentetc. Online analysis is performed on the client device to supportreal-time feedback in the application and the raw data is compressed anduploaded to the cloud to support more advanced analytics. An onlinecomponent may aggregate the bio-signals, and application data such astext and corrections initiated by the user to support the development ofadditional applications and algorithms.

Other Feedback Modalities

Feedback through a haptic or tactile feedback device, such as avibrotactile device may also be provided or modulated, such as in amobile phone, pager, vibrator, or other vibrating element in a device.Vibrotactile feedback can be directly proportional to brainwaves. Forexample, the more over the threshold of a certain band or state, themore intense the vibration, as controlled by the rules engine. Forexample, one's attention on a subject could be represented by beta bandactivity. As one's focus intensity increases, power of beta alsoincreases, and the intensity of the vibration of a motor inside a device(such as cell phone), could also increase.

Emotions or liking, disliking or anger could be represented as such. Forexample, if one is angry, the state could be transmitted to a tactileactuator, and the movements of the actuator could become more violent,stronger or faster. Likewise, calming could be represented as such aswell, and could be communicated.

Another example of vibrotactile feedback is someone thinking about afriend, and that friend's phone buzzing to share that the first personwas thinking about them. Another example may be a user configuring thephone such that every time he or she feels feelings of love oraffection, the user's child's phone or partner's phone or best friend'sphone can vibrate. Like emotion can be used to annotate text message, socan these communications be shared through vibrotactile feedback (e.g.an “I'm thinking about you” buzz).

Vibrotactile feedback can be proportional in intensity, respond indifferent frequencies, through different frequencies of pulsing buzzes,or other methods of expressing through vibrotactile feedback.

Vibrotactile feedback can also be in the form of music, the vibration ofa stereo system being transduced in such a way as it can be physicallyfelt by the user, such as in a bass shaker.

Additional feedback modalities may include: temperature, changingshapes, clothing, smell, and others. Emotional and state informationcaptured by brainwaves could also be shared via temperature, texture orsmell. For example, if someone is angry, a target device could heat up(change temperature) as a means of signalling and transmitting mood. Bymeans known in the art such as moving motors, actuators or solonoids, ormodulating substances like Nitol that change shape when a current isattached, the target device could be or have a flexible surface thatchanges shape based on response. For example, one could wear clothing,and when one is angry, the edges of the clothing could curl up.

General

It will be appreciated that any module or component exemplified hereinthat executes instructions may include or otherwise have access tocomputer readable media such as storage media, computer storage media,or data storage devices (removable and/or non-removable) such as, forexample, magnetic disks, optical disks, tape, and other forms ofcomputer readable media. Computer storage media may include volatile andnon-volatile, removable and non-removable media implemented in anymethod or technology for storage of information, such as computerreadable instructions, data structures, program modules, or other data.Examples of computer storage media include RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks(DVD), blue-ray disks, or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by an application, module, or both. Any suchcomputer storage media may be part of the mobile device, trackingmodule, object tracking application, etc., or accessible or connectablethereto. Any application or module herein described may be implementedusing computer readable/executable instructions that may be stored orotherwise held by such computer readable media.

Thus, alterations, modifications and variations can be effected to theparticular embodiments by those of skill in the art without departingfrom the scope of this disclosure, which is defined solely by the claimsappended hereto.

The present system and method may be practiced in various embodiments. Asuitably configured computer device, and associated communicationsnetworks, devices, software and firmware may provide a platform forenabling one or more embodiments as described above. By way of example,FIG. 46 shows a generic computer device 500 that may include a centralprocessing unit (“CPU”) 502 connected to a storage unit 504 and to arandom access memory 506. The CPU 502 may process an operating system501, application program 503, and data 523. The operating system 501,application program 503, and data 523 may be stored in storage unit 504and loaded into memory 506, as may be required. Computer device 500 mayfurther include a graphics processing unit (GPU) 522 which isoperatively connected to CPU 502 and to memory 506 to offload intensiveimage processing calculations from CPU 502 and run these calculations inparallel with CPU 502. An operator 507 may interact with the computerdevice 500 using a video display 508 connected by a video interface 505,and various input/output devices such as a keyboard 510, mouse 512, anddisk drive or solid state drive 514 connected by an I/O interface 509.In known manner, the mouse 512 may be configured to control movement ofa cursor in the video display 508, and to operate various graphical userinterface (GUI) controls appearing in the video display 508 with a mousebutton. The disk drive or solid state drive 514 may be configured toaccept computer readable media 516. The computer device 500 may formpart of a network via a network interface 511, allowing the computerdevice 500 to communicate with other suitably configured data processingsystems (not shown).

In further aspects, the disclosure provides systems, devices, methods,and computer programming products, including non-transientmachine-readable instruction sets, for use in implementing such methodsand enabling the functionality described previously.

Although the disclosure has been described and illustrated in exemplaryforms with a certain degree of particularity, it is noted that thedescription and illustrations have been made by way of example only.Numerous changes in the details of construction and combination andarrangement of parts and steps may be made. Accordingly, such changesare intended to be included in the invention, the scope of which isdefined by the claims.

Except to the extent explicitly stated or inherent within the processesdescribed, including any optional steps or components thereof, norequired order, sequence, or combination is intended or implied. As willbe will be understood by those skilled in the relevant arts, withrespect to both processes and any systems, devices, etc., describedherein, a wide range of variations is possible, and even advantageous,in various circumstances, without departing from the scope of theinvention, which is to be limited only by the claims.

What is claimed is:
 1. A system comprising: at least one computingdevice; at least one bio-signal sensor in communication with the atleast one computing device; at least one user input device incommunication with the at least one computing device; the at least onecomputing device configured to: present digital content at the at leastone computing device for presentation to at least one user; receivebio-signal data of the at least one user from the at least onebio-signal sensor, at least one of the at least one bio-signal sensorcomprising a brainwave sensor, and the received bio-signal datacomprising at least brainwave data of the at least one user; and modifypresentation of the digital content at the at least one computing devicebased at least partly on the received bio-signal data, at least onepresentation modification rule associated with the presented digitalcontent, and at least one presentation control command received from theat least one user input device.
 2. The system of claim 1 wherein thepresentation modifying comprises presenting at least a subset of thereceived bio-signal data based at least partly on the receivedbio-signal data.
 3. The system of claim 1 wherein the presentationmodifying comprises presenting at least one representation of the user'sbrain state based at least partly on the received bio-signal data, theat least one presentation modification rule, and the at least onepresentation control command.
 4. The system of claim 1 wherein thereceived bio-signal data comprises time-coded bio-signal data, and thedigital content presentation comprises time-coded presentation data, theat least one computing device configured to synchronize the time-codedbio-signal data to the time-coded presentation data at least partly bysynchronizing time stamps between the time-coded bio-signal data and thetime-coded presentation data, wherein the time-coded presentation datamay be used by the at least one computing device to determine at leastone presentation state of the presented digital content at a respectivetime stamp.
 5. The system of claim 4 wherein the presentation modifyingcomprises modifying presentation of the digital content based at leastpartly on a correspondence between the time-coded bio-signal data at atleast one time stamp with the time-coded presentation data at the atleast one time stamp.
 6. The system of claim 1 wherein the presentationmodifying comprises modifying presentation of the digital content basedat least partly on a bio-signal interaction classification profile. 7.The system of claim 6 wherein the bio-signal interaction classificationprofile is associated with the at least one user.
 8. The system of claim6 wherein the at least one computing device is configured to update thebio-signal interaction classification profile based at least partly onthe presentation control command.
 9. The system of claim 6 wherein theat least one computing device is configured to update the bio-signalinteraction classification profile based at least partly on bio-signalinteraction classification data received from at least one computerserver over a communications network.
 10. The system of claim 1 whereinthe at least one computing device is configured to update the at leastone presentation modification rule based at least partly on presentationmodification rule data received from at least one computer server over acommunications network.
 11. The system of claim 1 wherein the at leastone presentation control command indicates a target brain state, and thepresentation modification comprises presenting digital content that isassociated with the target brain state at the at least one computingdevice.
 12. The system of claim 1 wherein the presentation modificationcomprises the at least one computing device sharing at least a part ofthe presented digital content, or a link thereto, with at least oneother user on a content sharing server.
 13. The system of claim 12wherein the sharing comprises associated the received bio-signal datawith the shared presented digital content, and sharing the associatedreceived bio-signal data with the at least one other user on the contentsharing server.
 14. A system comprising: at least one computing device;at least one bio-signal sensor in communication with the at least onecomputing device; at least one user input device in communication withthe at least one computing device; the at least one computing deviceconfigured to: present digital content at the at least one computingdevice for presentation to at least one user; receive bio-signal data ofthe at least one user from the at least one bio-signal sensor, at leastone of the at least one bio-signal sensor comprising a brainwave sensor,and the received bio-signal data comprising at least brainwave data ofthe at least one user; determine at least one brain state of the user inresponse to the presented digital content, the brain state determiningbased at least partly on received bio-signal data; and associate thedetermined brain state with the presented digital content.
 15. Thesystem of claim 14 wherein the received bio-signal data comprisestime-coded bio-signal data, and the digital content presentationcomprises time-coded presentation data, the at least one computingdevice configured to synchronize the time-coded bio-signal data to thetime-coded presentation data at least partly by synchronizing timestamps between the time-coded bio-signal data and the time-codedpresentation data, wherein the time-coded presentation data may be usedby the at least one computing device to determine at least onepresentation state of the presented digital content at a respective timestamp.
 16. The system of claim 14 wherein the determined brain statecomprises at least one user emotion quantified by the at least onecomputing device in accordance with at least one bio-signal analysisrule.
 17. The system of claim 16 wherein the digital content comprisesadvertising content, and wherein the at least one computing device isconfigured to flag the respective advertising content as effective basedat least partly on the quantified emotion.
 18. The system of claim 17wherein the quantified emotion comprises a determined emotion intensity,and wherein the at least one computing device is configured to flag therespective advertising content as effective based at least partly on adetermined correspondence between the determined emotion intensity andan emotion response threshold associated with the determined brain statefor the respective at least one user.
 19. The system of claim 14 whereinthe at least one brain state determining is determined in real-time. 20.The system of claim 14 wherein the at least one computing device isconfigured to modify presentation of the digital content at the at leastone client computing device based at least partly on the at least onedetermined brain state and at least one presentation modification ruleassociated with the presented digital content.
 21. The system of claim20 wherein the at least one computing device is configured to modifypresentation of the digital content at the at least one client computingdevice based at least partly on at least one presentation controlcommand received from the at least one user input device.
 22. The systemof claim 20 wherein the at least one computing device is configured topresent a representation of the associated determined brain state. 23.The system of claim 14 wherein the brain state determining is based atleast partly on a bio-signal interaction classification profile.
 24. Thesystem of claim 14 wherein the at least one computing device isconfigured to receive communication content during a communicationsession established over a communications network with at least oneremote computing device; wherein the digital content presentingcomprises presenting the received communication content at the at leastone computing device for presentation to at least one user; and thedetermined brain state associating comprises associating the determinedbrain state with the received communication content.
 25. The system ofclaim 24 wherein the at least one computing device is configured totransmit the brain state associated with the received communicationcontent to the at least one remote computing device over thecommunications network.
 26. The system of claim 24 wherein thecommunication session comprises a videoconferencing communicationsession and the communication content comprises at least one videostream.
 27. The system of claim 24 wherein the at least one computingdevice is configured to receive brain state data from the at least oneremote computing device, the received brain state data representing atleast one brain state of at least one remote user of the at least oneremote computing device.
 28. The system of claim 27 wherein the receivedbrain state data comprises time-coded brain state data, and the receivedcommunication content comprises time-coded communication content, the atleast one computing device configured to synchronize the receivedtime-coded brain state data to the time-coded communication content atleast partly by synchronizing time stamps between the receivedtime-coded brain state data and the time-coded communication content,wherein the time-coded communication content may be used by the at leastone computing device to determine the brain state of the remote user ata respective time stamp of the received communication content.
 29. Thesystem of claim 24 wherein the at least one computing device isconfigured to modify presentation of the digital content at the at leastone client computing device based at least partly on the at least onedetermined brain state and at least one presentation modification ruleassociated with the presented digital content.
 30. The system of claim24 wherein the at least one computing device is configured to modifypresentation of the digital content at the at least one client computingdevice based at least partly on the at least one received brain stateand at least one presentation modification rule associated with thepresented digital content.
 31. The system of claim 24 wherein thepresentation modification comprises modifying appearance of a useravatar displayed at the at least one computing device.
 32. A systemcomprising: at least one computing device; at least one bio-signalsensor in communication with the at least one computing device; the atleast one computing device configured to: receive bio-signal data of atleast one user from the at least one bio-signal sensor, at least one ofthe at least one bio-signal sensor comprising a brainwave sensor, andthe received bio-signal data comprising at least brainwave data of theat least one user; determine at least one brain state of the user, thebrain state determining based at least partly on received bio-signaldata; and select digital content for presentation to the at least oneuser based on the determined at least one brain state of the user;present the selected digital content at the at least one computingdevice.
 33. The system of claim 32 wherein the digital content selectingcomprises selecting advertising content, and the determined at least onebrain state comprises at least one user emotion quantified by the atleast one computing device in accordance with at least one bio-signalanalysis rule.
 34. The system of claim 33 wherein the quantified emotioncomprises a determined emotion intensity, and wherein the at least onecomputing device is configured to flag the respective advertisingcontent as effective based at least partly on a determinedcorrespondence between the determined emotion intensity and an emotionresponse threshold associated with the determined brain state for therespective at least one user.
 35. A system comprising: at least onecomputing device; at least one bio-signal sensor in communication withthe at least one computing device; at least one user input device incommunication with the at least one computing device; the at least onecomputing device configured to: receive bio-signal data of at least oneuser from the at least one bio-signal sensor, at least one of the atleast one bio-signal sensor comprising a brainwave sensor, and thereceived bio-signal data comprising at least time-coded brainwave dataof the at least one user; determine at least one brain state of theuser, the brain state determining based at least partly on receivedbio-signal data, the determined at least one brain state comprisingtime-coded brain state data; recording time-coded user activity datarepresenting at least one user interaction with the at least onecomputing device; synchronize the time-coded brain state data to thetime-coded user activity data at least partly by synchronizing timestamps between the time-coded brain state data and the time-coded useractivity data; and determine the brain state of the at least one user ata respective time stamp of the respective user interaction based atleast partly on the time-coded user activity data.
 36. The system ofclaim 35 wherein the at least one user interaction comprises at leastone content creation activity by the user, and the recorded time-codeduser activity data represents the status of the at least one contentcreation activity at respective time stamps.
 37. The system of claim 35wherein the at least one computing device is configured to modifypresentation of digital content associated with the at least one userinteraction based at least partly on a correspondence between thetime-coded brain state data at at least one time stamp with thetime-coded user activity data at the at least one time stamp.
 38. Thesystem of claim 37 wherein a target brain state is associated at the atleast one computing device with encouraging at least one contentcreation activity, and wherein the modifying presentation of digitalcontent comprises presenting digital content that is associated at theat least one computing device with the target brain state.
 39. Thesystem of claim 38 wherein the system comprises at least one user inputdevice in communication with the at least one computing device; andwherein the target brain state is based at least partly on at least onepresentation control command received from the at least one user inputdevice.
 40. A method performed by at least one computing device incommunication with at least one bio-signal sensor, and at least one userinput device, the method comprising: presenting digital content at theat least one computing device for presentation to at least one user;receiving bio-signal data of the at least one user from the at least onebio-signal sensor, at least one of the at least one bio-signal sensorcomprising a brainwave sensor, and the received bio-signal datacomprising at least brainwave data of the at least one user; andmodifying presentation of the digital content at the at least onecomputing device based at least partly on the received bio-signal data,at least one presentation modification rule associated with thepresented digital content, and at least one presentation control commandreceived from the at least one user input device.